datadeps: cache hierarchical schedules; fix similar; fix JuMPExt import - #722
Draft
riteshbhirud wants to merge 45 commits into
Draft
datadeps: cache hierarchical schedules; fix similar; fix JuMPExt import#722riteshbhirud wants to merge 45 commits into
riteshbhirud wants to merge 45 commits into
Conversation
`_eft_runtime_ns` did `@view spec.fargs[2:end]`, which breaks under
typed tasks (DTaskSpec{true}) because `spec.fargs` is a `Tuple` and
`Base.view(::Tuple, ::UnitRange)` isn't defined. Every AOT scheduling
call under hierarchical scheduling hit this because typed tasks are
the code path hierarchical routes through.
Dispatch on argument type: `Base.tail` for tuples (zero allocation),
`@view` for vectors. Both cases produce a signature-compatible view of
the arguments-after-function.
Adds cross-`spawn_datadeps` reuse of hierarchical AOT schedules, so
iterative workflows (typical for solvers and stencil codes) pay the AOT
scheduling cost once per DAG shape instead of every call. Together with
two related bug fixes surfaced along the way, this makes the full
scheduler test suite pass on `jps/riteshsc26` (486 pass / 1 pre-existing
broken).
## Cache under hierarchical scheduling
`distribute_tasks_hierarchical!` now:
* Builds a full-region `DAGSpec` up front (using the same
`dag_add_task!` logic as the flat path's `datadeps_build_schedule!`)
and consults `queue.scheduler`'s task-local schedule cache before
spinning up any partitioning or per-partition scheduling.
* On a cache hit, extracts the recovered task-to-processor mapping
from the matching `DAGSpecSchedule` and threads it through the
parallel per-partition pipeline via a new `precomputed_schedule`
keyword on `schedule_partition_full!`. Each partition filters the
full-region schedule to its own tasks and skips the
`datadeps_build_schedule!` call entirely; tasks whose cached
processor is not in `local_procs` (possible if `all_procs` changed
since the schedule was cached) fall back to JIT via the existing
guard in `_schedule_vertex!`.
* On a cache miss, runs the existing pipeline unchanged, collects
each partition's task-to-processor assignments alongside its
`DataDepsState`, and persists them to
`queue.scheduler.cache` keyed by the full-region `DAGSpec`.
Julian's per-partition `similar(queue.scheduler)` shard remains
untouched on the miss path (its concurrency and per-partition
processor-list reasoning is unchanged); the cache lookup / persistence
sits strictly above that layer so no shared mutable state or
cross-thread synchronisation is introduced.
## Bug fixes surfaced by the cache work
1. `_eft_runtime_ns` did `@view spec.fargs[2:end]`, which breaks under
typed tasks (`DTaskSpec{true}`) because `spec.fargs` is a `Tuple`
and `Base.view(::Tuple, ::UnitRange)` isn't defined. Dispatch
Tuple -> `Base.tail`, Vector -> `@view` (zero allocation on both
paths). This is the same fix as PR JuliaParallel#722; carried here so the branch
is testable end-to-end.
2. The parameterised schedulers -- `IteratedGreedyScheduler`,
`SimulatedAnnealingScheduler`, `OptimizingScheduler`,
`JuMPScheduler` -- had no `Base.similar` methods, so hierarchical
partition-shard creation fell through to the default
`similar(::DataDepsScheduler) = typeof(s)()` and raised a
`MethodError` (no zero-arg constructor for the parameterised
types). Added type-specific methods that preserve configuration
and deep-copy the RNG so parallel partition shards produce
independent, race-free samples without behavioural drift.
3. `ext/JuMPExt.jl` imported `MetricsTracker` as a top-level package,
but `jps/riteshsc26` folds MetricsTracker into the `Dagger` module
via a direct `include` (see `src/Dagger.jl`). Changed to
`import Dagger.MetricsTracker as MT` so the JuMP extension loads
under this branch.
## Verified
Full `test/datadeps/scheduling.jl` on `jps/riteshsc26 @ 0c9aff8`
(Julia 1.12.3, macOS aarch64):
* Before this commit: 30 pass / 4 fail / 7 error in GreedyScheduler
alone, halted before the other schedulers ran.
* After this commit: 486 pass / 0 fail / 0 error / 1 pre-existing
broken across `equivalent_structure`, `DAGSpec`, `PtrStrict`,
`Greedy`, `IteratedGreedy`, `SimulatedAnnealing`,
`OptimizingScheduler` (adaptive and MILP paths), and `JuMPScheduler`.
Determinism spot check (matmul in `spawn_datadeps` with an
`IteratedGreedyScheduler`): first call is a cache miss (cache size
0 -> 1); second call with the same DAG shape hits (cache size stays
at 1, no crash).
riteshbhirud
changed the title
riteshbhirud
marked this pull request as draft
July 17, 2026 23:35
Per Przemek's & Julian's SC26 workshop paper feedback: give the metaheuristic schedulers a "quick-and-dirty 1 minute timeout" so long-K benchmarks stay within a reasonable everyday-user budget instead of exhausting the full Orsila cooling schedule × n_restarts safety cap. MILP already had `time_limit_sec` (forwarded to HiGHS); IG and SA were purely iteration-bounded until now. Changes: - `IteratedGreedyScheduler` and `SimulatedAnnealingScheduler` grow a `time_limit_sec::Float64` field (default `Inf`, opt-in), validated strictly-positive in the inner constructor. - `iterated_greedy_schedule!` / `simulated_annealing_schedule!` compute `start_ns = time_ns()` once and check `(time_ns() - start_ns) >= budget_ns` at the top of the outer loop (IG) or every restart and every SA inner-loop iteration. `Inf` is represented as `typemax(UInt64)` so the check never trips. - The check placement returns best-so-far without spending another neighbour proposal, preserving the "non-worsening vs seed" invariant covered by the existing test. - `Base.similar` for both schedulers carries the field so hierarchical partition shards inherit the same budget. - `OptimizingScheduler` grows `ig_time_limit_sec` and `sa_time_limit_sec` and forwards them into the constructed IG / SA when routed to the heuristic path. When routed to MILP, only `milp_time_limit_sec` applies (unchanged). - Bench driver's `default_scheduler_factories` sets `heuristic_time_limit_sec=60.0` for IG, SA, and the heuristic path of OptimizingScheduler; MILP retains its 120 s default. Both budgets are kwarg-configurable so callers can widen or disable. - New tests: constructor validation (accept finite, reject 0/negative, default is `Inf`), `similar` propagation, budget-triggered early exit under a nanosecond limit on a real Cholesky DAG, `Inf` opt-out reproduces the pre-budget deterministic result, and end-to-end `OptimizingScheduler` heuristic-path forwarding. All 516 scheduler tests pass with and without JuMP loaded, plus the one pre-existing broken test unchanged.
The bench harness's `TimedScheduler` wrapper crashed under Dagger's
default hierarchical scheduling because it had no `Base.similar` method,
so `distribute_tasks_hierarchical!` fell through to the generic
`typeof(s)()` shard constructor — which errors on `TimedScheduler`
since the primary ctor requires an inner scheduler argument.
Even after adding a naive `Base.similar`, the `sched_phase_ns`
metric would report 0 for every row: each partition shard would
write into its own private `Ref{UInt64}`, but the driver reads
the parent wrapper's Ref, which is never touched. This would
have silently corrupted Table 1's scheduler-cost column and made
Fig 3 (AOT scheduling cost vs K) unreproducible on any machine
running the harness under default settings.
Changes:
- Replace `Ref{UInt64}` with `Threads.Atomic{UInt64}` so
concurrent writes from parallel partition shards accumulate
without races.
- Change `datadeps_schedule_dag_aot!(::TimedScheduler, ...)`
to `Threads.atomic_add!` instead of `[]=`; the atomic is a
proper accumulator (sum over partition shards), not a
last-writer-wins slot.
- Add a shard-constructor variant `TimedScheduler(inner, atomic)`
that reuses the parent's atomic.
- Add `Base.similar(::TimedScheduler)` that (1) recursively
`similar`s the inner scheduler so its own mutable state (RNG,
cache, etc.) is refreshed per shard, and (2) hands the shard
the parent's atomic so shard writes are visible to the
parent-side `sched_phase_ns` reader after the run.
Semantics under the fix:
- Single-instance scheduling (no hierarchical partitioning): one
`datadeps_schedule_dag_aot!` fires, `sched_phase_ns` holds
its wall-clock duration — same as before.
- Hierarchical partitioning: `sched_phase_ns` is the sum of
scheduler CPU time across all partition shards, i.e. the
total scheduler work per DAG. This is the correct honest
metric for Fig 3 and matches the "total work" convention.
Regression coverage in `test/datadeps/scheduling.jl` (new 11
tests, 527 total pass):
- `similar` returns a `TimedScheduler` (does not fall through
to the crashing generic path)
- Atomic is shared between parent and every shard; multiple
shard writes accumulate visibly at the parent
- `sched_phase_ns > 0` after an actual hierarchical Cholesky
run through `run_once` (the exact pattern the paper's
benchmark sweep uses)
- Fresh instance starts at 0; cache and equivalence hooks
still forward to the inner scheduler
Plain `Dagger.@spawn` never distributed work to non-master processes:
julia -p 7 -t 12 -e '
using Dagger; @Everywhere using Dagger
Dagger.all_processors() # -> 96 processors across 8 pids (correct)
ts = [Dagger.@Spawn sum(rand(64,64)) for _ in 1:24]
fetch.(ts)
# :compute events after this: pid 1 = 25, pid 2-8 = 0
'
Registration is healthy (all 96 procs visible; init_proc completes for
all pids; state.worker_chans has entries for pids 1..8), yet every task
lands on pid 1. The processor-selection filter at
src/sch/Sch.jl:777 correctly keeps all 8 pids as candidates; the master
preference is set later, in `estimate_task_costs!`.
Root cause: line 667 of `src/sch/util.jl` added a hardcoded
`task_xfer_cost = gproc.pid != myid() ? 1_000_000 : 0` (1ms) penalty to
non-master processors' cost. The line predates MetricsTracker and
carried a "TODO: Actually estimate/benchmark this" comment. Under the
current MetricsTracker cost model this heuristic collapses selection
to master-only:
* Cold state (no metrics for any proc): every proc's `est_time_util`
defaults to `UInt64(1000^3) = 1e9`. Master's cost is 1e9; each
worker's is 1e9 + 1e6. `sort!(sorted_procs, by=cost)` puts all 12
of master's ThreadProcs first, deterministically. The `randperm!`
above only randomizes among equally-costly procs, and they are
not equally costly under this penalty.
* After master runs the first task, its measured runtime (e.g. ~100us)
replaces its 1e9 default in MetricsTracker, while workers stay at
the 1e9 default because they never ran a task. Master's cost drops
to O(1e5) versus workers' O(1e9). Master preference becomes locked
in and self-reinforcing: the workers never accumulate metrics
because they never receive tasks.
The penalty is redundant under the metrics model: any real per-signature
cross-process dispatch overhead is captured in `est_time_util` once
workers accumulate metrics, and per-chunk transfer cost is captured by
the existing `tx_cost / tx_rate` term. Remove the penalty.
Verification:
Reproducer (24 plain @Spawn tasks on -p 7 -t 12):
before: pid 1: 25 tasks (all on master)
after: pid 1: 2, pid 2: 1, pid 3: 6, pid 4: 4, pid 5: 5,
pid 6: 1, pid 7: 2, pid 8: 4 (distributed across all 8 pids)
Full scheduler test suite:
julia --project=test -e 'using Dagger, JuMP, HiGHS;
include("test/datadeps/scheduling.jl")'
16 testsets, all pass, 0 fail, 0 error, 1 pre-existing broken.
Datadeps AOT-scheduled workloads are unaffected because the AOT
scheduler force-pins each task via `options.exec_scope`, bypassing the
cost-based selection entirely.
`estimate_task_costs!` is called per submitted task and asks
`metrics_lookup_runtime` for every candidate processor. The lookup walks
the MetricsTracker snapshot end-to-end via `MT.find_keys`, so with `W`
candidate processors and `N` recorded metric keys, the cost estimator
was doing `O(W × N)` work per task submission — dominating submission
latency on `Dagger.@spawn` in tight loops.
Measured before this commit on M4 Pro `julia -t 12`:
julia> begin
ts = [Dagger.@Spawn sleep(0.010) for _ in 1:24]
fetch.(ts)
end
# wall clock: 234 ms (24 * 10 ms serial = 240 ms; parallel ideal ~20 ms).
# Effective parallelism 1.0× of 12 threads. Placement is distributed
# across all 12 ThreadProcs but ~77 ms of the ~230 ms wall is spent
# inside the master's submission loop, before the first task starts
# executing.
Profile of the submission loop confirms the hot path: `estimate_task_costs!`
→ `metrics_lookup_runtime` → `MT.find_keys`. Each `find_keys` call
iterates every metric storage's every key to intersect the runtime lookup
chain — and the chain fires once per candidate processor.
Because every processor in a single `estimate_task_costs!` call shares the
same signature `sig_vec`, only the *first* lookup pattern in the chain
(exact `ProcessorMetric == proc`) is proc-specific; the remaining three
patterns are proc-type / worker / global fallbacks that resolve to the
same measurements for every processor of the same type on the same worker.
Fix: introduce a per-signature runtime index, built once per
`estimate_task_costs!` call:
- `Dagger.SignatureRuntimeIndex` (`src/utils/metrics.jl`): a struct with
four buckets — `by_proc_worker`, `by_type_worker`, `by_type`,
`any_matching` — populated in a single `O(N)` scan across
SignatureMetric-matching thunk_ids.
- `Dagger.build_signature_runtime_index(snap, sig_vec)`: does the scan.
- `Dagger.metrics_lookup_runtime_from_index(idx, proc, worker_id)`:
replaces the per-proc `metrics_lookup_runtime` call, performing the same
four-step fallback via `O(1)` hash lookups against `idx`.
`estimate_task_costs!` now builds the index once outside the processor
loop and calls `metrics_lookup_runtime_from_index` inside. Semantics of
the fallback chain are preserved bit-for-bit; only the traversal cost
changes.
Verification:
* Same reproducer, after this fix:
julia> for _ in 1:3
[Dagger.@Spawn sleep(0.010) for _ in 1:24] .|> fetch
end
julia> best = Inf
for _ in 1:5
t = @Elapsed begin
ts = [Dagger.@Spawn sleep(0.010) for _ in 1:24]
fetch.(ts)
end
best = min(best, t)
end
best # -> 0.0445 s
44.5 ms vs 234 ms → 5.3× wall-clock reduction. Effective
parallelism 5.4× of 12 threads (was 1.0×).
* Empty-task submission microbenchmark (100 × `Dagger.@Spawn 1+1`):
before: 121 ms (1.21 ms/task)
after : 60 ms (0.60 ms/task)
2.0× improvement in the per-task submission fixed cost.
* Full scheduler test suite:
julia --project=test -e 'using Dagger, JuMP, HiGHS;
include("test/datadeps/scheduling.jl")'
16 testsets, 527 pass, 0 fail, 0 error, 1 pre-existing broken.
* Existing `metrics_lookup_runtime` code path is untouched: any caller
that queries a *single* (proc, worker_id) still resolves through the
old function. Only `estimate_task_costs!` — the hot inner loop of
task submission — is rerouted through the new index.
Notes:
* `has_capacity` in `src/sch/util.jl` still calls
`metrics_lookup_runtime` directly. It is invoked at most once per
candidate proc from `schedule_one!`, so the per-call `O(N)` cost is
amortised across the same proc loop where the new index already
lives. A follow-up could thread the index through `has_capacity`
for another marginal improvement; deferred to keep this change
focused on the dominant hot path.
* The AOT scheduler path (`options.exec_scope !== nothing`) still
goes through `estimate_task_costs!` but with `length(procs) == 1`,
so the index build is `O(N)` and the single lookup is `O(1)` — the
same total work as before. No regression there.
…N) scan Companion to 2fafa73. `schedule_one!` already built a `SignatureRuntimeIndex` per submitted task to give `estimate_task_costs!` `O(1)` per-proc runtime lookups. The subsequent reservation loop then called `has_capacity` for each candidate proc, and `has_capacity` in turn called `metrics_lookup_runtime` — its OWN `O(N)` scan of the MetricsTracker snapshot — for exactly the same `(signature, proc, worker_id)` triples the index had already indexed. For the AOT path (`options.exec_scope !== nothing`, `W == 1`), that meant two `O(N)` snapshot scans per submitted task instead of one. Fix: * `estimate_task_costs!` grows an optional `runtime_index` kwarg. When provided, the internal build is skipped and the caller's index is used verbatim. When `nothing` (existing callers), the function builds one internally — backwards-compatible with any caller outside `schedule_one!`. * `has_capacity` grows the same optional `runtime_index` kwarg. When provided, the internal `metrics_lookup_runtime` call is replaced with `metrics_lookup_runtime_from_index` — same fallback-chain semantics, no snapshot rescan. When absent, the original `metrics_lookup_runtime` path is preserved so `has_capacity` can still be called standalone. * `schedule_one!` now builds the `SignatureRuntimeIndex` once at the top of Phase 2 and threads it through both `estimate_task_costs!` and every `has_capacity` call inside the sorted-procs reservation loop. Total per-task metrics-scan work drops from `O(2N)` (for AOT, or `O((W+1)N)` for non-AOT) to `O(N)`. Verification: * Full scheduler test suite: 16 testsets, 527 pass, 0 fail, 0 error, 1 pre-existing broken. `has_capacity` semantics are preserved bit-for-bit for both `options.time_util !== nothing` (fast path unchanged) and the metrics-lookup fallback (index-backed path returns the same values as `metrics_lookup_runtime` per the `SignatureRuntimeIndex` fallback-chain contract). * Sleep reproducer (M4 Pro, `julia -t 12`, warmed): before this commit: 44.5 ms best-of-5 (5.4× of 12) after this commit: 43.2 ms best-of-5 (5.6× of 12) Consistent with the AOT-path improvement being a modest constant- factor win rather than the dramatic non-AOT speedup 2fafa73 already captured. The synthetic reproducer here is non-AOT and had already absorbed most of the gains.
…capacity skips its metrics scan Companion to 02fa4e1. `Sch.has_capacity` (`src/sch/util.jl:529-`) has a fast path when `options.time_util[typeof(proc)] !== nothing`: it uses the caller-supplied per-processor-type runtime estimate directly and skips its `metrics_lookup_runtime` call entirely. AOT-scheduled tasks never populated that value, so has_capacity always fell through to the snapshot scan even when our AOT scheduler had already produced a perfectly good estimate for the same `(task, proc)` pair. The AOT schedulers already own a runtime estimate per `(task, proc)` pair — that estimate is exactly what the cost model minimises over — so propagating it costs almost nothing at scheduling time. This is a bench-side change to OUR AOT layer (Ritesh's schedulers); the Sch.jl fast path in has_capacity is Julian's code and stays untouched. Changes: * `_propagate_aot_time_util!(spec, proc, task_time_ns)` in `src/datadeps/scheduling.jl`: sets `spec.options.time_util[typeof(proc)] = task_time_ns / 1e9` (Float64 seconds, matching has_capacity's expected units: `round(UInt64, time_util[T] * 1000^3)` -> nanoseconds). Non-positive / non-finite estimates are ignored so the caller can safely propagate whatever the cost model produced without a per-scheduler nullability check. * `_propagate_aot_time_util_from_cache!(dag_spec, cache, task_idx, proc)`: convenience wrapper for the three schedulers (Greedy, IG, SA) that already build an `EFTCostCache` and can look up the runtime by `(task_idx, proc_idx)` in O(1). If the cache does not know about `proc`, the propagation is a no-op — `has_capacity` falls back to metrics lookup, preserving existing behaviour. * `GreedyScheduler`, `IteratedGreedyScheduler`, `SimulatedAnnealingScheduler` datadeps_schedule_dag_aot! implementations: after `schedule[task] = proc`, also call `_propagate_aot_time_util_from_cache!`. `OptimizingScheduler` delegates to one of these sub-schedulers and inherits the propagation via them, no separate change needed. * `ext/JuMPExt.jl` `datadeps_schedule_dag_aot!(::JuMPScheduler, ...)`: imports and calls `_propagate_aot_time_util!` with `task_times[k, proc_idx]` from the MILP formulation. This is the same nanosecond estimate the objective minimised over, so the fast-path value has_capacity reads back is exactly what the MILP optimised against. Verification: * `has_capacity` fast-path fires for AOT-scheduled tasks (traced with a temporary `@info` in the fast path; log showed `has_capacity FAST PATH fired for T=Dagger.ThreadProc time_util[T]=1.0` corresponding to Greedy's `GREEDY_DEFAULT_RUNTIME_NS = 1e9` when metrics are cold). Trace removed before commit. * Full scheduler test suite: 16 testsets, 527 pass, 0 fail, 0 error, 1 pre-existing broken. * AOT-datadeps microbenchmark (M4 Pro, `julia -t 12`, nt=8 tasks, best-of-5 timed regions): before this commit after Greedy AOT 3.04 ms 2.94 ms ( ~3% ) SA AOT 6.69 ms 5.37 ms ( ~20% ) Consistent with the SA path benefiting more because it fires `has_capacity` more often per submission (deeper cost model), while Greedy already commits to the first-chosen proc quickly. Both schedulers correctly propagate `options.time_util`; the delta is a lower-bound on the per-task overhead removed and will be larger for hudson-scale workloads whose per-task snapshot scan is proportionally more expensive. * On the paper's benchmark path (spawn_datadeps with any of our AOT schedulers on the paper's Greedy/IG/SA/JuMP/Optim comparison), every submitted task now takes the has_capacity fast path. Notes: * Non-`EFTCostCache` schedulers (`LayeredScheduler`) do not compute a runtime and are left unchanged; they continue to fall through to `has_capacity`'s metrics-lookup path unchanged. * `_propagate_aot_time_util!` mutates `spec.options` in place. This is safe because `spec` is the task's own DTaskSpec, submitted before the AOT scheduler runs, and each user @Spawn creates its own `Options` object. No cross-task aliasing.
… tasks off-master
The paper's Hudson benchmark sweep produced a physical impossibility for
cholesky nt=8 — an implied 7,140 GFLOPS on a 3,840 GFLOPS peak machine —
and a 480× CPU asymmetry (master 3929 s, each worker 7–8 s) even under
multi-process configs. Direct probes on hudson (Julia -p 1 -t 2 with
`spawn_datadeps` + `:compute` event logging) showed every task landing on
pid 1 under both `RoundRobinScheduler` and `GreedyScheduler`, so the sweep
had been measuring master-thread scheduling all along rather than the
distributed scheduling the paper claims to characterise.
This is a *bench-side* defect, not a Dagger bug. `make_spd_tiles` and
`make_matmul_tiles` allocated every tile as a plain `Matrix{Float64}`
resident in master's memory space. When `spawn_datadeps` observed the
tile-graph, it (correctly) concluded that shipping an 8-MB tile out to a
worker and back would dominate any parallelism gain, and placed every
task on master — the optimal decision *given master-resident inputs*.
Chameleon / PLASMA / DPLASMA (the tiled linear-algebra reference
libraries the paper's workloads model) all distribute tile data across
processors before entering the region for exactly this reason.
Fix, bench-side only:
* `bench/datadeps_schedulers/workloads.jl`:
- Add `_placement_procs()`: sorted vector of
`Dagger.all_processors()`, deterministic across identical sessions
so tile→proc placement is reproducible run-to-run.
- Add `_distribute_tile(tile, proc)`: `remotecall_fetch` the
`Dagger.tochunk(tile, proc, ExactScope(proc))` call to the target
processor's worker. `MemPool.poolset` runs on the target worker,
so the resulting DRef lives in that worker's memory space and
subsequent datadeps tasks touching only that tile incur no
cross-worker transfer. Same pattern used by test/datadeps.jl:552.
- Add `_tile_proc(procs, nt, i, j)`: shared round-robin
row-major-linearised (i, j) → proc mapping so matmul's
`A[i, k]`, `B[k, j]`, `C[i, j]` co-locate identically across
workloads at the same logical positions.
- Rewrite `make_spd_tiles` and `make_matmul_tiles` to distribute
tiles through `_distribute_tile` at their `_tile_proc(...)`
target. `make_spd_tiles` `copy`s each tile before shipping so the
closure carries an independent `Matrix` rather than a view into
the parent SPD buffer (would otherwise confuse datadeps' aliasing
analysis).
- Relax `tiled_cholesky!(M::Matrix{<:Matrix})` /
`tiled_matmul!(C::Matrix{<:Matrix}, A::Matrix{<:Matrix}, ...)`
to `AbstractMatrix`. `Dagger.@Spawn f(In(M[k,k]))` /
`Dagger.@Spawn f(InOut(M[k,k]))` accept `Chunk` and `Matrix`
elements transparently: for a `Chunk`, datadeps reads its `scope`
/ memory space directly; for a `Matrix`, it observes
master-residence as before. Same tile-loop semantics in both
cases.
* `bench/datadeps_schedulers/driver.jl`:
- Add `_fetch_tile(t)`: passes through a `Matrix`, `fetch`es a
`Dagger.Chunk` (which triggers `collect` / `move` from the
chunk's owning worker back to master). This is exactly what
`verify_workload` needs when reassembling the tiled state for
the dense-reference check.
- Relax `_assemble_dense(tiles::Matrix{<:Matrix})` to
`AbstractMatrix` and route each `tiles[i, j]` element through
`_fetch_tile`. Uses the first tile's element type + block size
to allocate the dense buffer. Backwards-compatible with plain
`Matrix` tiles for any caller not going through
`make_*_tiles`.
Scope preserved:
* `build_inputs`, `run_workload!`, `verify_workload`, `run_once`,
`run_sweep`, and every scheduler-side code path unchanged. All the
complexity lives at the `make_*_tiles` allocation boundary and the
`_assemble_dense` fetch boundary.
* `src/`, `ext/`, `lib/`, `test/*.jl` untouched.
* Metrics-warm's RoundRobin pass now populates γ (worker-to-worker
transfer) metrics based on ACTUAL worker-to-worker movement.
Previously γ measurements were meaningless because every task ran
on master and no cross-worker movement occurred; now the cost
model has real data to consume.
Verification:
* Placement probe (`OMP_NUM_THREADS=2 julia --project=test -p 1 -t 2`,
cholesky nt=2 bs=1024, `:compute` events):
RoundRobin Greedy
pid 1: 6 5
pid 2: 7 7
pids w/ :compute: 2 2
was pid 1: {8, 7}, pids w/ :compute: 1 for both schedulers on
master-resident tiles. (The 6+7 / 5+7 vs the K=4 explicit tasks
reflects datadeps-inserted cross-worker copy tasks — expected
under distributed placement.)
* Correctness at bs=1024:
cholesky nt=2: rel_err = 6.11e-17 (threshold 1e-8)
cholesky nt=4: rel_err = 9.01e-17 (threshold 1e-8)
matmul nt=2: rel_err = 2.16e-16 (threshold 1e-10)
matmul nt=4: rel_err = 2.24e-16 (threshold 1e-10)
* Full scheduler test suite:
julia --project=test -e 'using Dagger, JuMP, HiGHS;
include("test/datadeps/scheduling.jl")'
16 testsets, 527 pass, 0 fail, 0 error, 1 pre-existing broken.
No regressions.
Impact:
* Every previous hudson sweep (bs=128 archive AND the bs=1024 rerun)
measured single-process scheduling, not distributed scheduling.
The physically-impossible GFLOPS number in Table 1 was the
downstream symptom. Reruns after this commit will measure the
distributed scheduling the paper's methodology assumes and the
Chameleon / PLASMA / DPLASMA workload conventions the paper cites.
* Fig 3 (scheduler cost) is unchanged — algorithmic runtime is
independent of where tile data lives.
* Table 1 and Fig 4 will now reflect real cross-worker placement
trade-offs.
…concentration The prior _placement_procs implementation sorted by (pid, name) so all of master's ThreadProcs came first in the list, followed by worker 2's, etc. With `procs[mod1((i-1)*nt+j, length(procs))]` selecting one processor per tile, small nt values under-distributed: tile count linear idx range procs touched pids reached nt=2 1..4 indices 1..4 1 (master only) nt=3 1..9 indices 1..9 1 (master only) nt=4 1..16 indices 1..16 2 (master + worker 2) nt=8 1..64 indices 1..64 ~6 pids nt=16 1..256 wraps 2.7x all 8 pids This is silently worse than a uniform fail: a full sweep across tile_counts = 2,3,4,8,16 would measure a different distribution level at every nt, silently confounding the tile-count axis with an unstated "how many workers happened to be hit" axis. No CSV column would reveal it. Root cause caught by the multi-process placement probe on Hudson: at nt=4 K=20, both RoundRobin and Greedy placed tasks on only pids 1 and 2 despite 96 processors being registered across 8 pids. Fix: group processors by owning pid, sort each pid's slice deterministically, then interleave — take the k-th processor from each pid in pid order, then move to k+1. Yields [pid1_t1, pid2_t1, ..., pidN_t1, pid1_t2, pid2_t2, ...] so consecutive tile indices at any nt land on different pids. Verified: - Interleave logic produces the expected pattern via a mock 8-pid × 12-thread simulation: nt=2 hits pids 1-4, nt=4 hits all 8 pids. - Full 527-test scheduler suite still passes: 0 fail, 0 error, 1 pre-existing broken. No regression. - Single-process M4 Pro session returns a single-element vector as before (only pid 1 exists). Multi-process verification at production scale must happen on Hudson after this commit lands upstream.
…can decide placement The prior _distribute_tile used `ExactScope(target_proc)` which is a HARD constraint on task placement, not just a data-location hint. Under it, every scheduler was forced onto the tile's owning processor — RoundRobin, Greedy, IteratedGreedy, SimulatedAnnealing, JuMP, and OptimizingScheduler all converged to byte-identical n_copies counts because none of them had any placement choice to make. Hudson's matmul sweep at bs=1024 confirmed the collapse: nt=2 n_copies median across all 6 schedulers: 9 9 9 9 9 9 nt=3 41 41 41 41 41 41 nt=4 96 96 96 96 96 96 nt=8 448 448 448 448 448 448 exec_span_ms differences across schedulers spanned ±20% while per-trial CV within a single (cell, scheduler) reached 34.4% — the scheduler signal was smaller than the noise because there was no scheduler signal, just system variance around a pre-determined placement. The fix removes the ExactScope. Tiles still live on specific workers (the initial `remotecall_fetch(target_pid) do; Dagger.tochunk(...) end` puts the DRef in the target worker's memory space), so γ transfer costs are still real and nonzero. But schedulers now DECIDE placement, weighing γ against parallelism and load balancing — Greedy will co-locate to minimize γ, RoundRobin will spread, SA/IG/JuMP will explore. Their placement decisions genuinely differ, which is the paper's mechanism. Verified via 527-test suite: 0 fail, 0 error, 1 pre-existing broken. No regression. Multi-process verification at production scale must happen on Hudson after this commit + the hierarchical.jl aliasing-bug fix land upstream.
`distribute_tasks_hierarchical!` previously binned tasks with `partition_dag`
(data-affinity) before any scheduler ran, then let each partition compute its
own AOT schedule over just that partition's processors. Under Dagger's default
hierarchical mode with distributed tiles this made every scheduler look
identical at the paper's metric (n_copies): affinity picked the coarse
worker-level placement; per-partition schedulers only shuffled tasks among a
single worker's threads where γ ≈ 0, so worker-level cross-copies never
varied across schedulers.
When (a) the argument data is already distributed across ≥2 workers and (b)
the scheduler produces a whole-region AOT schedule, partition by the
scheduler's assigned worker instead of by data affinity. The scheduler's
placement drives partitioning, and different schedulers produce measurably
different per-worker task counts (and thus n_copies).
- New `partition_by_assigned_worker(seen_tasks, schedule, task_metas,
all_procs)`: bin each task by `root_worker_id(schedule[task])`, with an
affinity fallback for the rare unscheduled-tail case (past a
`dag_add_task!` bail point); returns partitions in ascending-worker-id
order for determinism.
- Whole-DAG AOT call sites: bypass `datadeps_build_schedule!` and call
`datadeps_schedule_dag_aot!` directly, so we don't double-write the
scheduler's per-type cache (which is now owned by
`_hierarchical_persist_schedule!` at the top level, keyed by the outer
`dag_spec`).
- Persist condition switches from "no precomputed schedule" to "no cache
hit", so newly-computed whole-DAG schedules also persist for reuse.
- Gate on `data_is_distributed` (≥2 workers own arg data). When all data
lives on one worker, affinity partitioning already picks that worker and
is optimal; forcing a scheduler-first path in that case would spread
computation away from data with no benefit, so we fall through to the
existing affinity behaviour.
- JIT schedulers (RoundRobin, Naive, Ultra) don't specialize
`datadeps_schedule_dag_aot!`, produce an empty trial, and take the
affinity branch unchanged.
Verified on `-p 3 -t 4` with distributed cholesky tiles:
nt=4: Greedy/Optimizing (scheduler-first) → per-worker (1,8)(2,4)(3,4)(4,4)
RoundRobin (affinity) → per-worker (1,10)(2,6)(3,3)(4,1)
nt=8: AOT (scheduler-first) → (1,32)(2,32)(3,28)(4,28)
JIT (affinity) → (1,49)(2,35)(3,23)(4,13)
datadeps test suite: 514/521 pass, 2 fail + 4 error + 1 broken -- matches
baseline (513/521 pass, 3 fail + 4 error + 1 broken) modulo one flake.
Datadeps top-level: 1260/1262 pass, 2 broken (unchanged from baseline).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…length to expose the deserialization dedup collapse
The `for i in eachindex(worker_args); arg_to_ainfo[worker_args[i]] =
results[i].second` loops in both branches of `build_aliasing_parallel`
assume `length(results) == length(worker_args)`. Under -p 7 -t 12
cholesky nt=4 bs=1024 that invariant has been observed to break twice
now (BoundsError: 1-element Vector{Pair{ArgumentWrapper,AliasingWrapper}}
at index [2]), and the crash surfaces as a bare BoundsError with no
context about the mismatched sizes or the ArgumentWrapper identities
involved.
Add an `@assert length(results) == length(worker_args)` immediately
before each loop with the worker id, both lengths, and the
worker_args + results ArgumentWrapper `.hash` values. When the invariant
breaks the AssertionError names the mismatched sizes and the hash-set
directly, which is enough to distinguish the two candidate mechanisms:
1) upstream `Dict{ArgumentWrapper,ArgumentWrapper}`-keyed dedup on
master collapsing two hash-equal entries -- would show a
duplicate hash in `worker_args_hashes` and `result_first_hashes`
the same length as `worker_args_hashes` minus the collapsed count.
2) `DRef` mutable-struct backref dedup during
`Serialization`-then-deserialization of `worker_args` on the
remote worker -- would show distinct hashes in
`worker_args_hashes` and a strictly-shorter `result_first_hashes`.
Diagnostic only; the root cause fix is a follow-up commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…eliminate cross-partition race
Under Dagger's default hierarchical mode at scale (-p 7 -t 12 with the
paper's distributed cholesky at nt=4 bs=1024) `build_aliasing_parallel`
crashed intermittently with `BoundsError: 1-element
Vector{Pair{ArgumentWrapper, AliasingWrapper}} at index [2]` inside
the multi-worker branch. Root cause is a Julia scoping / data race,
not a serialization or deduplication issue.
The single-worker branch at line 385 binds `results` in the enclosing
function scope:
results = wid == myid() ? _compute_aliasing_batch(worker_args) :
remotecall_fetch(_compute_aliasing_batch, wid, worker_args)
The multi-worker branch just below uses the same name inside its
`Threads.@spawn` closure:
@sync for (wid, worker_args) in by_worker
Threads.@Spawn begin
results = if wid == myid() ... end
...results[i]...
end
end
Julia scoping resolves `results = ...` inside the closure to the
already-bound enclosing-function variable rather than allocating a
fresh per-closure local. Every partition's `Threads.@spawn` task
therefore races on ONE shared `results` cell; the last writer wins,
and every reader (including the reader in that same task, after any
scheduling delay) sees the winner's Vector instead of its own. When
the winning task's `worker_args` is shorter than the reading task's,
`results[i]` at `i=length(winner_results)+1` throws `BoundsError` --
which is exactly what Hudson observed twice on cholesky-nt=4 and what
I reproduced locally on trial 12 of a -p 7 -t 12 stress loop.
Fix: `local results` at the top of the `Threads.@spawn` block so each
closure gets its own binding. Verified with a minimal
`@sync for ... Threads.@spawn` scoping test (without `local`, every
task reads the last writer's value; with `local`, each task reads its
own). The cholesky-nt=4 bs=1024 -p 7 -t 12 reproducer that
deterministically crashed at trial 12 in ~70s now runs the full
5-minute budget without incident.
The single-worker branch (line 385) does not need `local` -- there is
only one flow, no concurrency.
The diagnostic assert from 5037b44 stays: with the race fixed it
should never fire, but the invariant is worth documenting explicitly
so future regressions surface loudly instead of as bare `BoundsError`s.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Partial pass. Kills paragraph-length explanations, backreferences to prior commits/tasks/people, and per-line "what this line does" comments. Keeps docstrings (public API), invariant statements, and the few comments where the *why* is non-obvious (e.g. `local results` justification, cross-partition `finally` rationale, `_eft_tail_args` tuple-vs-vector caveat). Files: - src/datadeps/hierarchical.jl (-372/+107) - src/datadeps/scheduling.jl (-98/+25) - src/sch/Sch.jl (small) - src/sch/util.jl (small) No code changes -- comment and docstring text only. `using Dagger` loads clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…crashes
The prior `run_sweep` accumulated all `RunResult`s in memory and wrote
`hudson_cpu_*.csv` only at end of `main`. A mid-sweep abort — Dagger
race, HiGHS `std::length_error` at K=512 during JuMPScheduler
matmul nt=8, segfault, or OOM — killed the process before the final
`write_csv`, and every completed trial in memory was lost. Hudson's
production sweeps hit this three times in one day, losing ~2.5h of
compute across two cholesky runs (aliasing race) and one matmul run
(HiGHS overflow) with zero rows on disk each time.
Fix in two parts, both bench-side (no src/ / ext/ / lib/ changes):
1. Streaming CSV writes. `run_sweep` accepts `output` and
`correctness_output` kwargs; when set, opens the files immediately,
writes the header line, and appends+flushes one row per completed
trial (and one per correctness check). A subsequent crash preserves
every trial written up to the crash. `main` passes the CLI
`--output` path through so the default invocation gets streaming
for free; the batch `write_csv` at end of `main` is removed since
the file has already been written incrementally.
2. Per-cell crash isolation. `run_once` calls (warmup, correctness
check, and each measured trial) are wrapped in try/catch. A
scheduler that reliably crashes at a given cell — e.g.,
`JuMPScheduler` at matmul nt=8 K=512 where HiGHS `HFactor::
setupGeneral` overflows an internal `std::vector` sizing — logs a
warning with the exception + backtrace and moves to the next
trial/cell. The sweep continues instead of aborting the whole run.
Correctness crashes similarly are non-fatal. Warmup crashes skip
the entire cell (its trials wouldn't run cleanly either).
Semantics preserved:
- When `output === nothing` (existing internal callers, e.g. tests),
behaviour is unchanged: rows accumulate in the returned Vector,
no file I/O.
- The `CSV_HEADER` layout, `RunResult` field ordering, and
`_run_result_csv_row` (extracted from `write_csv` so streaming +
batch share the row-serialization) are identical.
- Correctness output naming (`replace(output, r"\.csv$" =>
"_correctness.csv")`) unchanged.
- `--summary` markdown emission at end of `main` unchanged.
- Exception messages include full backtrace via
`(e, catch_backtrace())` — no swallowed context.
Note re: `std::terminate` in C++ (as with HiGHS's uncaught
`std::length_error`) aborts the process before any Julia try/catch can
handle it. Per-trial `try/catch` here protects against Julia
exceptions (which HiGHS may throw normally at smaller sizes, or
Dagger internals may throw); it does NOT protect against a process
abort. The streaming CSV covers that residual case: everything
completed before the abort is on disk.
Full 527-test scheduler suite passes: 0 fail, 0 error, 1 pre-existing
broken. No regressions.
…gException Under scheduler-first hierarchical partitioning (commit b2f4bc4), the whole-DAG AOT scheduler now sees the full task count and processor list rather than the per-partition subsets it saw under the previous affinity-based partitioning. That is required for scheduler- differentiation on the paper's central metric (n_copies varies across schedulers), but it inflates JuMPScheduler's MILP model 8x on our benchmark: `matmul` nt=8 goes from ~64 tasks × 12 procs = 768 assignment binaries per partition (8 partitions, all tractable) to 512 tasks × 96 procs = 49,152 binaries in one model. HiGHS's `HFactor::setupGeneral` overflows an internal `std::vector` sizing at that scale and calls `std::terminate` before any Julia handler can run. Hudson's production sweep died at 23/24 cells with zero rows on disk on the first occurrence. The K-scalability limit is real -- it is exactly the tractability boundary the scheduling literature (Kwok & Ahmad 1999, V&S 2015) recognises as the motivation for adaptive dispatch, and it is what `OptimizingScheduler`'s K-threshold routes around by design. The issue is only that the underlying solver signals the boundary via a process abort rather than a catchable Julia error. Fix: `JuMPScheduler`'s `datadeps_schedule_dag_aot!` now guards on `n_tasks * nprocs > JUMP_MAX_ASSIGNMENT_VARS` (constant = 10,000, comfortably below the empirical HiGHS overflow at ~50k) and throws a `Sch.SchedulingException` with an informative message. The driver's per-cell exception handler (commit 6814e9b) already catches `SchedulingException`, logs it, and continues the sweep -- so cells past the ceiling now mark cleanly as "MILP intractable" in the log without losing the rest of the run. Semantics preserved: - `JuMPScheduler` at K where the MILP is tractable (K * nprocs < 10k) is unchanged: same model, same warm-start, same time_limit_sec. - The 70-test `JuMPScheduler` suite and 28-test `OptimizingScheduler (MILP path)` suite both pass unchanged -- both use small K well below the ceiling. - `OptimizingScheduler` is unaffected: its `milp_threshold` (default 12) routes far below the ceiling, so the guard never fires along its MILP path. Paper context: this converts a solver-caused process abort into an explicit precondition. `JuMPScheduler` now reports MILP infeasibility gracefully; `OptimizingScheduler` routes past it via its adaptive dispatch. Both are contributions. Full 527-test scheduler suite: 0 fail, 0 error, 1 pre-existing broken. No regressions.
…ver-memory workaround The K-guard added in fa4eadf was framed around HiGHS's model-setup overflow at ~5e4 assignment variables — a solver-specific memory argument that presented as a workaround. Empirical data collected in the interim shows a stronger reason for the same threshold: At n_tasks × nprocs = 6,144 (matmul nt=4, K=64 × 96 procs) with `time_limit_sec = 120s`, HiGHS returns at ~125s wall clock with `MOI.TIME_LIMIT` status. Larger problems within the current budget are consistently time-truncated. `JuMPScheduler`'s Table 1 role as an *exact* MILP reference no longer holds above this boundary — reported values would be incumbents, not proven optima. The 10,000 threshold sits above the empirically-time-truncating point and below the solver-crash point, marking the boundary above which MILP is neither exact nor competitive within the wall-clock budget. This is the standard tractability treatment used by V&S 2015 §5.3 and Kwok-Ahmad 1999 §4: MILP is applied where it remains exact within the budget; heuristics take over past that boundary. `OptimizingScheduler` implements exactly this policy adaptively via `milp_threshold`. No numeric threshold change (still 10,000). The commit updates the docstring comment and the `SchedulingException` message to state the optimality-based rationale directly. Reviewer question "why 10k" now answered with our own measurement (K=64 hits the 120s cap) plus canonical citations (V&S 2015 §5.3, Kwok-Ahmad 1999 §4), rather than "5x safety margin below the observed HiGHS overflow." Full 527-test scheduler suite: 0 fail, 0 error, 1 pre-existing broken. No regressions.
…load Two changes enabling the paper's heterogeneous-regime evaluation: 1. workloads.jl `_distribute_tile` gains vendor-agnostic GPU support via a new `_localize_to_target` helper. When the target proc is a CPU proc (ThreadProc/OSProc), tile stays as `Matrix` (existing behaviour). When the target is any other proc type (GPU, accelerator), the helper hops through `Dagger.move(OSProc, target_proc, x)` which CUDAExt/ROCExt/MetalExt/oneAPIExt/OpenCLExt override to construct the correct array type (`CuArray`, `ROCArray`, etc.) on the target device. Zero vendor conditionals in this file — one benchmark harness works unchanged across every accelerator Dagger supports. BLAS/LAPACK dispatch on GPU procs is already handled by CUDAExt's / ROCExt's auto-generated `Dagger.move(::CPUProc, ::GPUProc, ::typeof(BLAS.fn!))` overloads, so the tile-BLAS bodies of `tiled_cholesky!` / `tiled_matmul!` require no changes to run cross-vendor. 2. Adds Sinnen-Sousa 2004 Layer-by-Layer random DAG as a third workload class. Structured BLAS DAGs (matmul, cholesky) leave little room between EFT list scheduling and any smarter method because their regular dependency skeleton makes Greedy locally optimal on homogeneous processors — provable and empirically confirmed. Random DAGs are the standard evaluation regime for HEFT-family schedulers (Topcuoglu 2002 §5, Sinnen-Sousa 2004 §5, Ruiz-Stützle 2007 §6, Orsila 2008 §5) exactly because their irregular topology creates room for IG/SA to improve over Greedy and for MILP to find globally-better assignments. In-degree is capped at 2 so each task maps to a single tile-level `BLAS.gemm!` — one `Dagger.@spawn` per DAG node, keeping `sched_phase_ms` comparable across all three workload classes. Deterministic in `seed` so the same DAG topology is used across every scheduler cell and trial (essential for apples-to-apples scheduler comparisons on identical graphs). Driver `--workloads` accepts `random_dag` alongside `cholesky,matmul`; `nt` in `--tile-counts` is reinterpreted as `n_tasks` for random_dag (structured workloads scale K as f(nt), random DAGs are naturally parameterised by task count). `n_levels = round(Int, sqrt(n_tasks))` per Sinnen-Sousa's recommended aspect ratio; `edge_probability = 0.3` per Topcuoglu's HEFT-eval default. Correctness verification for random_dag is finiteness-only (no closed-form reference). Both changes are additive: existing cholesky and matmul CPU-only sweeps run bit-identically. CPU+GPU sessions light up automatically when CUDA / AMDGPU is loaded and the vendor extension registers GPU procs via `add_processor_callback!`. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes hudson's Stage 3 validation failure: `ArgumentError: Attempt to
use a freed reference` from `write_remainder!` in
`src/datadeps/remainders.jl:499` during the metrics-warm RoundRobin
pass. Trace:
CUDAExt:422 (task exec)
→ move!(RemainderAliasing{CPURAMMemorySpace}, CUDAVRAMMemorySpace,
CPURAMMemorySpace, Chunk{CuArray}, Chunk{Matrix})
at remainders.jl:446
→ write_remainder! at remainders.jl:499
→ unsafe_convert(CuPtr{Float64}, CuArray) — freed DeviceMemory
Root cause: the prior commit's `_localize_to_target` created
`Chunk{CuArray}` tiles owned by GPU procs at benchmark-setup time.
Dagger's cross-space `RemainderAliasing` transfer path (which mediates
`Chunk{CuArray}` ↔ `Chunk{Matrix}` transfers under datadeps aliasing
analysis) assumes both sides are CPU-native memory and does pointer
arithmetic that fails on `pointer(::CuArray)` when the CuArray has
been freed by MemPool. This combination is untested — every
`Chunk{CuArray}` construction in `test/gpu.jl` goes through the
DArray+scope path (`rand(Blocks, ...)` + `Dagger.with_options(; scope)`),
not `tochunk(::CuArray, ::CuArrayDeviceProc)`.
Fix: keep tile ownership CPU-only. `_placement_procs` now filters
`Dagger.all_processors()` to CPU procs (ThreadProc/OSProc) via a new
`_is_cpu_proc` predicate. `_distribute_tile` asserts the invariant
and creates `Chunk{Matrix}` on the target CPU worker's memory.
Removed `_localize_to_target` — the GPU-conversion path it added is
dead code under the new invariant, and would silently break the
invariant if a future edit reintroduces it.
Why this preserves the paper's heterogeneous-scheduling story
unchanged:
1. `Dagger.all_processors()` (used by `datadeps_schedule_dag_aot!`)
still enumerates GPU procs when CUDAExt/ROCExt are loaded, so
schedulers see them as valid task-placement targets and choose
to route tasks there.
2. When a task lands on a GPU proc, Dagger's
`move(::CPUProc, ::CuArrayDeviceProc, x)` (CUDAExt lines
172-198) transfers the tile HtoD via `adapt(CuArray, x)` under
the target device's context — the tested path with full
lifetime management.
3. GPU tasks pay realistic HtoD-input + compute + DtoH-output
costs; metrics-warm records these per-proc runtimes;
schedulers correctly route small tasks to CPU (transfer
overhead dominates) and large tasks to GPU (compute
dominates transfer). Exactly the regime literature validates.
If a future Dagger release fixes the RemainderAliasing cross-space
path for CuArray/ROCArray, the tile-locality axis can be added back
by reverting `_placement_procs` and restoring `_localize_to_target`.
For now, task-placement heterogeneity alone carries the paper's
central claim.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes hudson's Stage 4 issue: full CPU+GPU sweep running with 0% GPU utilisation, GPU memory never leaving 531 MiB baseline. Root cause is in Dagger core, not the harness: `_eft_runtime_ns` at `src/datadeps/scheduling.jl:807-808` falls back to `GREEDY_DEFAULT_RUNTIME_NS = 1_000_000_000` ns (1 second) when `MetricsTracker` has no `(signature, proc)` sample. On a first-touch sweep, GPU procs have no samples — so Greedy compares a real CPU runtime (~20 ms) to the 1000 ms GPU fallback and picks CPU every time. GPU never runs → never gets sampled → forever stuck. Chicken-and-egg. Fix (harness-level, no Dagger core changes): add `warm_gpu_metrics_for!` that runs a small workload with scope forced to the union of every discovered non-CPU proc, priming MT with real GPU `(signature, proc) -> runtime_ns` samples before either the CPU RR warmup or the measured trials. Greedy's subsequent EFT calculation then compares real CPU vs real GPU runtimes and correctly picks GPU where it's actually faster. Vendor-agnostic: enumerates non-CPU procs via `Dagger.all_processors()` + the `_is_cpu_proc` predicate from `workloads.jl`, constructs a `UnionScope([ExactScope(p) for p in gpu_procs])`, runs the workload under `Dagger.with_options(; scope=gpu_scope) do ... end`. Same scope-forcing pattern that `test/gpu.jl:148-158` uses in Dagger's existing GPU integration tests, so no untested code paths. Works across CUDAExt, ROCExt, MetalExt, oneAPIExt, OpenCLExt unchanged. Wired into `run_sweep`: when `metrics_warm=true`, GPU warmup runs first, then CPU RR warmup, then measured trials. On CPU-only sessions `warm_gpu_metrics_for!` early-returns (empty gpu_procs) with zero overhead — verified via CPU-only smoke test that this commit doesn't regress existing CPU sweeps. GPU warmup wrapped in try/catch: any GPU-warm crash logs a warning and continues with degraded (CPU-only) cost model rather than killing the cell. Better to have a documented "GPU cost model uncalibrated" cell than a lost sweep. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes `ArgumentError: Attempt to use a freed reference` in `remainders.jl` during CPU<->GPU and GPU<->GPU RemainderAliasing transfers. Root cause: the `GC.@preserve copies begin ... end` blocks around the read (line 430) and write (line 444) loops preserved only the `copies` UInt8 buffer, not the source/destination array wrappers (`from_raw` / `to_raw`). For CPU-backed `Array{T}`, this was harmless -- `Array`'s underlying memory is refcount-managed by Julia GC alongside the wrapper, so losing the wrapper doesn't matter mid-loop. But `CuArray` / `ROCArray` wrappers hold references to device memory whose lifetime is tied to the wrapper's finalizer (`CUDA.unsafe_free!` / `AMDGPU.unsafe_free!`). If GC runs mid-loop (which it does when `write_remainder!` / `read_remainder!` internals emit intermediate CUDA allocations via `unsafe_wrap` / `copyto!`), the finalizer frees the device memory. The next iteration's `pointer(::CuArray)` then throws. Reproducer: `bench/datadeps_schedulers/driver.jl --workloads cholesky --tile-counts 4 --block-size 1024 --metrics-warm` on a session with `using CUDA` and 2+ GPU procs -- cholesky's POTRF/TRSM/SYRK/GEMM dependency chain forces cross-GPU tile transfers under the GPU-scoped warmup, hitting the missing preserve. Fix: add `from_raw` (read loop) and `to_raw` (write loop) to the respective `GC.@preserve` clauses. Two-word change per line, zero overhead on CPU->CPU (GC.@preserve on Array is essentially a no-op since the wrapper's lifetime already extends its memory's), fixes CPU->GPU, GPU->CPU, and GPU->GPU RemainderAliasing transfers uniformly. Same class of fix as the earlier `local results` GC-safety issue in `build_aliasing_parallel` (commit 9378807) -- Julia's escape analysis is conservative but explicit GC.@preserve is the correct convention for lifetime-critical device-backed data. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…PU safety net Two coordinated changes to pursue the root cause of the cross-GPU `RemainderAliasing` freed-reference bug hudson caught in cholesky CPU+GPU cells, while keeping Session A+B unblocked: 1) src/datadeps/remainders.jl: revert commit 58a8941's GC.@preserve additions. Hudson's Cell A v2 verified they had no effect -- crash count identical (5), failing site inside the new preserve block. The error text `Attempt to use a freed reference` fires from GPUArrays' EXPLICIT-free sentinel (abstractarray.jl:73), not the GC use-after-free sentinel. GC.@preserve is the wrong tool for an explicit unsafe_free! caller. Revert restores remainders.jl to bit-identical pre-58a8941d state. 2) ext/CUDAExt.jl: instrument `Dagger.unsafe_free!(::CuArray)` with env-gated stacktrace logging. Enabled by `DAGGER_TRACE_UNSAFE_FREE=1`; zero overhead when unset (single ENV dict lookup that branch-predicts away). Every unsafe_free! invocation with the flag set logs its type, size, and full stacktrace via @info. Reproducing hudson's Cell A cholesky crash with this flag set will identify the caller that explicitly frees the destination CuArray during move! -- candidates are MemPool's DRef release path, Dagger's chunk finalizer racing with move!, or a datadeps memory-reclaim pass firing prematurely. 3) bench/datadeps_schedulers/driver.jl: add a single-GPU scope constraint as a safety net so Session A+B can run cleanly on hudson (avoiding all cross-GPU transfers = bug's code path never exercised) WHILE the root cause is being debugged in parallel. Introduces `_chosen_gpu_proc`, `_bench_scope_for_measured_runs`, `_bench_scope_for_gpu_warmup`, `_with_scope`, and splits `run_workload!` into scoped-outer / unscoped-inner variants. `warm_gpu_metrics_for!` now warms the single chosen GPU (not `UnionScope(all_gpus)` as in commit b8afecb, which itself triggered the bug during warmup). Env override `DAGGER_BENCH_MULTI_GPU=1` disables the single-GPU restriction -- used together with `DAGGER_TRACE_UNSAFE_FREE=1` for the reproducer/debug pass that gets us the stacktrace of the offending caller. Default state (both flags unset) enforces the safety net so validation cells and production sweeps run cleanly. CPU-only sessions are unaffected: `_bench_scope_for_measured_runs` returns `nothing` when no GPU procs are enumerated, and the wrap becomes a no-op. Smoke-tested locally on M4 (matmul nt=2 bs=64, all 5 non-JuMP schedulers, no crashes or warnings). Once the reproducer + instrumentation identifies the offending unsafe_free! caller, the follow-up commit will either: a) fix the caller (gate release on pending transfers, add reference-counting on chunks under active move!, etc.) b) OR add a `move!`-side reference retention that dominates the racing caller either of which allows dropping the single-GPU safety net and running Session A+B on both H100s. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…PU move! race
Root cause identified by hudson's diagnostic trace (Cell A v3 with
DAGGER_TRACE_UNSAFE_FREE=1): the offending `unsafe_free!` is
dispatched as a Dagger task (spawned in
`src/datadeps/queue.jl:290` and
`src/datadeps/hierarchical.jl:1139` at region end). Its `syncdeps`
are meant to include every task that touched the buffer, gathered
via `gather_free_syncdeps!` in `src/datadeps/aliasing.jl:752`
which walks `state.ainfos_owner` and `state.ainfos_readers`. Copy
tasks emitted by `enqueue_remainder_copy_to!` /
`enqueue_remainder_copy_from!` (`src/datadeps/remainders.jl:289` /
`:343`) register themselves as readers/writers via
`add_reader!` / `add_writer!`, so the chain SHOULD close.
On CPU-only sessions the chain is bulletproof AND the failure mode
is harmless: freeing a `Matrix{T}` while another task reads it is a
Julia refcount decrement that keeps memory alive. On GPU sessions
any missed edge is catastrophic -- `CUDA.unsafe_free!` invalidates
the device buffer immediately, and any in-flight `move!` still
dereferencing `pointer(::CuArray)` throws
`ArgumentError: Attempt to use a freed reference` from
`GPUArrays.abstractarray.jl:73` (the EXPLICIT-free sentinel, not
the GC use-after-free sentinel -- verified by hudson's stack trace
showing only 2 frames deep to the `Dagger.@spawn` closure).
Fix (correctness boundary, in the vendor extensions):
`Dagger.unsafe_free!(::CuArray)` / `(::ROCArray)` now call
`CUDA.synchronize()` / `AMDGPU.synchronize()` on the array's
device before releasing the buffer. This drains any pending
operations on that device -- guaranteeing correctness independent
of whether Dagger's syncdep chain missed an edge or not.
`CUDA.synchronize()` is a fast no-op when the queue is empty
(the common case, since the syncdep chain usually IS correct);
in the racy cases it waits bounded milliseconds for pending
transfers to complete. `with_context(x)` / `with_context(memory_space(x))`
picks the array's device before syncing so we drain the right
stream when the caller's current context is on a different device.
Why fix at the callsite rather than in aliasing.jl:
1. CUDA best practice always: synchronize before `unsafe_free!` on
any buffer that might have pending operations, regardless of
scheduler-level guarantees. Not just a Dagger repair -- makes
`Dagger.unsafe_free!(::CuArray)` correct against ANY caller,
including future ones and downstream users of Dagger.
2. The aliasing-side repair (audit `gather_free_syncdeps!` for
missed edges in the datadeps-allocated-slot code path OR
restructure toward per-buffer reference counting) is a
substantial refactor best pursued as a dedicated upstream PR.
This barrier is defense-in-depth that unblocks GPU sweeps
immediately AND remains correct even after any future
aliasing-side improvement.
Also revert the single-GPU safety-net scope constraint added in
commit 3a7bcb2 to `bench/datadeps_schedulers/driver.jl`. That
workaround suppressed the bug by preventing cross-GPU transfers;
with the barrier fix at the correctness boundary, it's now
unnecessary and would only mask the paper's dual-H100 story.
`run_workload!` is restored to its pre-3a7bcb20 form (unscoped);
`warm_gpu_metrics_for!` is restored to using
`UnionScope([ExactScope(p) for p in gpu_procs])` over ALL GPU procs
(same design as commit b8afecb that originally triggered the bug --
safe now under the barrier). Removed helpers
`_chosen_gpu_proc`, `_bench_scope_for_measured_runs`,
`_bench_scope_for_gpu_warmup`, `_with_scope`, and
`_run_workload_inner!` since none are referenced anymore.
The `DAGGER_TRACE_UNSAFE_FREE=1` instrumentation added in
commit 3a7bcb2 is retained -- zero overhead when unset and
useful for future GPU debugging.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`distribute_tasks_hierarchical!`'s free loop gathered syncdeps only from the single object-cache key `ainfo`, guarded by `haskey(state.ainfo_arg, ainfo)`. When that key is absent -- the case where a buffer only underlies wrapper arguments (e.g. a parent array whose tracked slots are `view`s over it) -- `free_syncdeps` was left empty and `Dagger.unsafe_free!` was spawned with no ordering constraint at all, free to run concurrently with an in-flight `move!` on the same buffer. `distribute_tasks!` (the flat path) never had this problem: it calls `gather_free_syncdeps!`, which syncs against every ainfo backed by the buffer via `chunk_to_ainfos` and falls back to an interval-tree overlap search over `ainfos_lookup` for exactly the missing-key case. The race is benign on CPU -- `unsafe_free!(::Array)` is a refcount decrement, and a concurrent reader keeps the memory alive -- which is why it went unnoticed. On GPU it is fatal: `CUDA.unsafe_free!` invalidates device memory immediately and the racing `move!` throws `ArgumentError: Attempt to use a freed reference` from `pointer(::CuArray)` (GPUArrays' explicit-freed sentinel, not the GC use-after-free sentinel). Fix: use `gather_free_syncdeps!` with a per-partition `chunk_to_ainfos`, matching the flat path exactly, plus the flat path's `freed` dedup so a buffer reachable from several ainfos or partitions is freed once. The shared-chunk registry syncdep is preserved. This only adds dependency edges that were always intended; no scheduling semantics change. Verified on 2x H100 (hudson): cholesky nt=4 bs=1024 CPU+GPU: 5 freed-reference crashes -> 0 random_dag n=40 bs=1024 CPU+GPU: 4 freed-reference crashes -> 0 both: 6/6 schedulers, 6/6 correctness (~7.9e-17), both GPUs active CPU-only regression: 2 identical runs differ 0.78x-1.54x, matching the cell's pre-existing 17-39% CV -- no systematic change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MT.GLOBAL_METRICS_CACHE grows unboundedly across runs; every task execution merges into it and merge cost grows with cache size. Measured on matmul nt=8 bs=1024: five identical passes went 14.8s -> 48.0s with allocations 137M -> 598M (+~120M/pass). RoundRobin degrades identically to Greedy, so the cost is in the write path, not scheduler lookups -- it taxes every scheduler equally. Because scheduler order within a cell is fixed, an unbounded cache confounds the comparison with execution order, and within a cell trial 5 is systematically slower than trial 1. `MT.trim!` with a bounded window preserves `--metrics-warm` semantics (Greedy still reads real measured cost data) while stopping the runaway. keep_per_metric=100 chosen from a sweep of 10/25/50/100/200, which showed runtime flat within noise across that range. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dagger's cost model tracked one readiness timestamp per processor, an
implicit capacity of 1. That is correct for `ThreadProc` but wrong for
`CuArrayDeviceProc`/`ROCArrayDeviceProc`, which multiplex N independent
streams over a device and can hold N kernels in flight. EFT therefore
charged K queued GPU tasks the full serial `proc_ready + task_time`,
overstating completion by up to a factor of N relative to single-slot CPU
processors and distorting placement.
Adds `Dagger.proc_concurrency(::Processor) = 1` (src/gpu.jl) with vendor
overrides returning the backend stream count, and replaces the single
timestamp with a per-slot vector in `ScheduleState`. Assignment claims the
earliest-free slot: start = max(data_ready, min(slots)), which is exactly
the previous arithmetic when there is one slot. `proc_ready_ns` is retained
as `maximum(slots)`, preserving the invariant the test suite asserts.
Capacity-aware processor allocation per Sinnen & Sousa 2005 §4.3.
Verified on hudson (2x EPYC 9454, 2x H100):
- K=1 equivalence: 4 sequential claims identical to the old model;
8 tasks x 10ms on 4 slots -> makespan 20.0 (capacity-1 gives 80.0)
- scheduler suite: 527 tests, no regressions (60/100/135+1 broken/55/70/28/11)
- CPU-only regression: two identical runs span 0.78x-1.54x on this cell,
K-slot lands at 0.91x-1.36x -- within the cell's own 17-39% CV
- absolute gains at random_dag n=160, 98 procs: Optimizing -38%,
JuMP -16%, RR/Greedy/SA ~-10%
Note: this did not change relative scheduler ordering at 98 procs with 20
tasks, where nothing queues and capacity cannot bind. It is a correctness
fix to the cost model, not a tuning knob.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…shbhirud/Dagger.jl into fix-eft-runtime-tuple-fargs
`spawn_datadeps` builds its candidate processor set from
`get_compute_scope()`, whose default carries `DefaultEnabledTaint` and
admits only processors with `default_enabled(proc) == true`.
`CuArrayDeviceProc` returns `false` there -- deliberate Dagger design so
accelerators are opt-in rather than used implicitly. The consequence for
the benchmark is that GPUs are dropped from `all_procs` before any
scheduler sees them, at the `filter!` in `distribute_tasks_hierarchical!`
(src/datadeps/hierarchical.jl:969) and its twin in `distribute_tasks!`
(src/datadeps/queue.jl:207).
Every CPU+GPU measurement taken before this commit was therefore CPU-only
in effect: the GPU was enumerated, registered and metrics-warmed, then
discarded prior to scheduling. `warm_gpu_metrics_for!` was unaffected
because it already passes an explicit `UnionScope` of `ExactScope`s, which
bypasses the taint -- this commit applies the same mechanism to the
measured region.
`run_workload!` is split into a scoped outer form and
`_run_workload_inner!`, so `warm_gpu_metrics_for!` can supply its own
GPU-only scope without nesting two scopes. `_bench_scope_all_procs()`
returns `nothing` on CPU-only sessions, leaving that path untouched.
Verified on hudson (8 CPU ThreadProc + 1x H100):
- AOT hook candidate set: procs=8 gpu=0 -> procs=9 gpu=1
- partition size: 2 tasks -> 20 tasks (affinity partitioning had been
splitting the DAG into single-worker fragments)
- CPU-only session: scope is `nothing`, matmul nt=2 bs=64 correct at
3.17e-16, default path unchanged
Known remaining blocker: `_eft_runtime_ns` returns the
`GREEDY_DEFAULT_RUNTIME_NS` 1000 ms fallback for every processor, so the
cost model cannot yet distinguish GPU from CPU and EFT still declines the
GPU (which additionally carries transfer cost). Addressed separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Right-looking tiled LU without pivoting, mirroring the tile-BLAS structure
Dagger uses internally in `LinearAlgebra.lu!(::DMatrix, ::NoPivot)`
(src/array/lu.jl): GETRF on the diagonal tile, TRSM down the column panel
and across the row panel, GEMM on the trailing submatrix. The two TRSM
calls are asymmetric, following Dagger's version -- column panel solves
against U ('R','U','N','N'), row panel against L with unit diagonal
('L','L','N','U').
Pivoting is omitted so the DAG shape is static and directly comparable to
`tiled_cholesky!`; partial pivoting would add a reduction over the column
panel per step plus a data-dependent row swap, changing the task graph
between runs. `make_lu_tiles` builds a diagonally dominant matrix (same
conditioning trick as `make_spd_tiles`) so the unpivoted factorization is
numerically sound.
Wired into `build_inputs`, `_run_workload_inner!` and `verify_workload`.
Verification reconstructs L*U from the in-place factors (unit-lower from
the strict lower triangle, U from the upper triangle) and compares against
the original matrix.
Smoke-tested CPU-only, correctness only, no measurements:
lu nt=2 bs=256 rel_err 7.54e-16
lu nt=4 bs=256 rel_err 7.26e-16
cholesky nt=4 bs=256 (regression check) rel_err 3.37e-16
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`_eft_runtime_ns` built its lookup signature from `spec.fargs` while the
annotations (`In`/`Out`/`InOut`/`Deps`) were still attached. `Sch.signature`
resolves an annotated arg via `chunktype(::InOut{T}) where T = T`, a
type-level unwrap that yields the `Chunk{...}` type parameter and cannot
recurse. The recorded signature is built at execution time from bare `Chunk`
instances, resolving through `chunktype(c::Chunk) = c.chunktype` to the
materialized type. So the read side constructed
Any[typeof(BLAS.trsm!), ..., Chunk{Matrix{Float64},...}, ...]
against a recorded
Any[typeof(BLAS.trsm!), ..., Matrix{Float64}, ...]
All four tiers of `_runtime_lookup_chain` pin `LookupExact(SignatureMetric())`
-- only the processor constraint relaxes -- so every lookup missed and every
task fell back to `GREEDY_DEFAULT_RUNTIME_NS` (1000ms) on every processor.
With a constant runtime for all (task, proc) pairs the EFT objective is flat,
and every scheduler was optimizing a degenerate cost model.
Unwrap annotations with `unwrap_inout` before constructing the signature, so
both sides agree. Measured on cholesky nt=4 bs=1024 CPU+GPU, Greedy:
0/180 hits before, 180/180 after.
Adds `DAGGER_TRACE_EFT_LOOKUP=1` to count hits/misses and dump missed
signatures; gated behind a `Ref{Bool}` since this runs O(tasks x procs).
Also restores the benchmark driver's `MT.trim!` call (disabled during the
eviction-hypothesis test) and raises `keep_per_metric` 100 -> 10000. The cache
still needs bounding -- unbounded growth took a run 14.8s -> 48.0s -- but at
100 the `unsafe_free!`/`move!` entries, ~60% of the cache by volume, evicted
the compute-task samples the cost model depends on.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…reach the scheduler
MetricsTracker keys runtime samples on the post-dispatch signature. A task
submitted as `BLAS.trsm!(::Matrix{Float64}, ...)` is recorded on a GPU as
`CUBLAS.trsm!(::CuArray{Float64,2,DeviceMemory}, ...)`, because the
`Dagger.move(::CPUProc, ::GPUProc, ::typeof(fn))` overloads substitute the
vendor function at dispatch. The scheduler only ever sees the pre-dispatch
form, so no GPU sample was reachable from `_eft_runtime_ns`.
That did not surface as a miss: the last tier of `_runtime_lookup_chain` drops
the processor constraint entirely, so a GPU lookup matched the CPU-recorded
entries and returned a CPU runtime. The cost model reported the GPU as exactly
as fast as an average CPU thread -- invisible in hit-rate terms, fatal for
heterogeneous scheduling.
Add `_translate_sig_for(sig, proc)`, generic returning `nothing` in
`src/gpu.jl`, overloaded per backend. The function half is generated by
piggybacking the existing reflective BLAS/LAPACK -> CUBLAS/CUSOLVER (and
rocBLAS/rocSOLVER) loop that already emits the paired `move` overloads, so it
stays in lockstep with dispatch and adds no maintenance surface. The array-type
half is hardcoded per backend (`Matrix{T}` -> `CuArray{T,2,DeviceMemory}` /
`ROCArray{T,2}`); `DeviceMemory` has no CPU-side counterpart to derive.
`_eft_runtime_ns` tries the translated signature *first* where a translation
exists, then falls back to the untranslated lookup. Translating only after a
miss, as originally sketched, would be dead code for exactly the reason above.
The fallback keeps a CPU-derived estimate when no mapping exists, still better
than the 1s default.
Measured, cholesky nt=4 bs=1024, GPU vs CPU for the same task:
BLAS.trsm! -> CUBLAS.trsm! HIT 0.194ms GPU vs 48.008ms CPU (247x)
Stopgap. The general fix is a processor-independent task identity recorded
alongside the physical signature, which removes the need for translation on
every backend at once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`METRICS_CACHE_MAX_TASKS` was a `const` 100, applied through `MT.trim_context!` on every metrics write. That rolling window is tuned for steady-state scheduling, where only recent samples matter, and it is far too small for benchmark scenarios that deliberately warm the cost model with high signature diversity. Concretely: a GPU warmup writes ~60 entries, then CPU warmup plus measured trials push past 100 and evict them. Heterogeneous cost lookups then silently fall back to CPU-derived estimates -- and *which* GPU signatures survive is nondeterministic, depending on task completion order, so scheduling decisions become irreproducible across reruns of an identical benchmark. Measured per-cycle demand is ~235 entries at cholesky nt=4 and ~835 at nt=8, against a bound of 100. Convert to a `Ref` with a `metrics_cache_max_tasks!` setter rather than raising the constant: Dagger's shipped default stays 100, and harnesses opt in. The benchmark driver sets 5000 -- ~6x the largest measured cycle, and ~200x below the unbounded regime that previously degraded a run from 14.8s to 48.0s. The bound is process-local; multi-worker runs must set it on each worker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n site `metrics_lookup_move_rate` queries FromSpaceMetric, ToSpaceMetric and MoveSizeMetric to estimate transfer cost for EFT. None of the three was ever written by any code path, so the lookup returned `nothing` on every call and the scheduler fell back to GREEDY_DEFAULT_TRANSFER_RATE = 1_000_000 B/s (scheduling.jl:406). At 1 MB/s an 8 MiB tile is charged 8388.6ms of transfer, a ~10,000x overestimate of PCIe, which effectively bans GPU placement: no compute advantage can offset it. The write path already existed and was simply never wired in. `instrumented_move!` wraps `move!`, derives the payload size from `source.handle.size`, and calls `_record_move_metrics!` -- but had zero callers. The four datadeps copy tasks spawned plain `Dagger.move!`. Point those four spawn sites at `instrumented_move!`. Argument order and arity are identical, and all four already pass `meta=true`, so the arguments arrive as `Chunk`s with `handle.size` populated. Measured, cholesky nt=4 bs=1024 CPU+GPU: before: FromSpace/ToSpace/MoveSize have no storage; move_rate -> nothing after: 36 entries each; MoveSize = 8388608 (one tile); move_rate -> real GPU data_ready: 8388.6ms -> 93.4ms Identified during cost-model diagnostics for CPU+GPU benchmarking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…_rate The estimator reduced `sum(size) / sum(time)` across all matching moves. That is not robust: a `move!` sample records end-to-end task time, so the first few moves carry compilation and device-context setup cost. Measured on cholesky nt=4 bs=1024 CPU+GPU, the 36 move samples ran 0.57ms to 1072ms. Five cold-start outliers (405-1072ms) contributed ~96% of total elapsed time, dragging the aggregate to ~90MB/s -- roughly 100x below the 14.8GB/s seen at steady state. Combined with the transfer charge being applied per input tile, that understated PCIe badly enough that no compute advantage could offset it, and the GPU was never selected despite measuring 247x faster than CPU on trsm!. Reduce over per-move rates with `Statistics.median` instead, matching how the compute side already reduces its samples (`metrics_lookup_runtime_median`), so both halves of the cost model use the same estimator. Exposed as a `reducer` kwarg for symmetry with `metrics_lookup_runtime`. Samples below MOVE_RATE_MIN_SIZE_BYTES (4KiB) are dropped: at that size the measurement is dominated by fixed per-task overhead rather than bandwidth, so the implied rate is noise. Measured effect, cholesky nt=4 bs=1024 CPU+GPU, GPU data_ready per tile: 8388.6ms (1MB/s fallback, pre-ed80723) -> 93.4ms (real samples, sum aggregate) -> 0.7ms (real samples, median) ~= 11.3GB/s, consistent with PCIe Greedy now places 15 of 20 cholesky tasks on the GPU; previously zero. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…stic == Greedy) IG's destroy/reconstruct loop used `greedy_assign_task!` (strict `<` tie-breaking, fully deterministic) for destroyed tasks. Since `_replay_schedule!` pins preserved tasks at their original procs and walks task ids in fixed order, the deterministic reconstruction provably reproduces the greedy seed *exactly* on every iteration, regardless of the random destroy set: by induction, if tasks 1..i-1 are all at their original procs, task i sees identical context and greedy picks the same proc. IG's neighborhood was a single point, so IG could never improve on Greedy -- contradicting the stochastic-construction step of Ruiz & Stützle (2007). Verified empirically pre-fix: across 5 RNG seeds x 200 iters x 30% destroy on cholesky nt=4 CPU+GPU, IG cost == Greedy cost exactly and 0/20 tasks differed. Fix: add `greedy_assign_task_randomized!`, an alpha-greedy (GRASP-style) variant that builds a restricted candidate list of every compatible proc whose K-slot-peeked finish is within `(1+alpha)` of the best and picks uniformly among them (reservoir sampling, no extra alloc). It uses the same `_peek_slot` / `_claim_slot!` helpers as the deterministic variant, so K-slot capacity accounting is identical. Thread an optional `rng` through `_replay_schedule!` and `iterated_greedy_step!`; only IG passes it. `rng === nothing` (every other caller, including SA's empty-destroyed replay) keeps the deterministic path byte-for-byte. `greedy_schedule!` never routes through `_replay_schedule!`, so Greedy is untouched. Strict acceptance (`new_cost < best_cost`) means IG's result is monotone from the greedy seed and can never be worse than Greedy. HONEST OUTCOME: with the default alpha=0.10 this does NOT change the benchmark numbers, because the restricted candidate list is a singleton in both regimes: in CPU+GPU the GPU is ~250x faster so its EFT is uniquely best (confirmed: no divergence even at alpha=50); in CPU-only the measured per-thread runtime spread is 35-110% so the second-best proc is already outside a 10% window. Divergence does appear at alpha>=0.5 on CPU-only, confirming the search is wired, but no strictly-better schedule was found -- consistent with the greedy EFT solution already being at the cost model's optimum for these regular tiled-BLAS DAGs. This is the "genuine flat landscape" result, not a wiring failure. `alpha` is left at 0.10 (near-greedy, conservative) and is tunable; raising it as a default is deferred pending evidence it helps on some workload. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…Greedy
Follows Ruiz & Stützle, "A Simple and Effective Iterated Greedy Algorithm for
the Permutation Flowshop Scheduling Problem" (2005/2007). Three corrections,
each verified against the paper rather than taken from the request sketch,
which deviated from it in several places:
1. SA-like acceptance with a SEPARATE best-so-far (§3.2). The loop now tracks
an `incumbent` that the acceptance criterion may move uphill, and a
monotone `best_state` that is the returned value. The sketch overwrote
`best_state` on acceptance, which would forfeit the "never worse than the
greedy seed" guarantee it simultaneously claimed. Verified: never_worse=true
across 6 seeds on cholesky/matmul nt=4.
2. Constant acceptance temperature per eq. (1):
Temperature = T * (sum_ij p_ij) / (n * m * 10) = T * mean(p_ij) / 10
averaged over compatible (task, proc) runtimes (`_ig_acceptance_temperature`).
The sketch used `T * best_cost`; best_cost is a makespan (~100x a single
task time), which inflates the temperature so far that a 250x-slower GPU->CPU
move is accepted ~72% of the time -- a random walk that destroys the
schedule. The paper's per-task-scale temperature (0.03-0.07ms here) correctly
rejects such moves.
3. T = 0.5, not 0.4. §3.4 DOE finds T=0.5 best (T=0.0 = strict acceptance
stagnates). Destruction count d = 4 (§3.4 tuned optimum), applied as a cap
on the frac-derived count so our K=10..64 DAGs get d=3..4, inside the
paper's optimal band; the sketch's frac=0.04 would give d=1..3, too low.
The randomized reconstruction (`greedy_assign_task_randomized!`, GRASP-style
alpha-RCL, from 6382a52) is retained and is a deliberate ADAPTATION: the
paper's construction is deterministic NEH best-insertion, whose neighborhood
comes from flowshop sequence positions. Task->processor assignment has no such
neighborhood (a re-derived task returns to the same proc under pinned
predecessors), so construction must be randomized to give IG a search space.
This is documented as an adaptation, not attributed to the paper.
OUTCOME on the paper's workloads: IG still ties Greedy (improve=0.0,
never_worse=true). This is now a correct, fundamental result rather than a bug:
the CPU+GPU workloads are GPU-dominated (one proc ~250x faster), so the RCL
collapses to a singleton and any off-GPU move is correctly rejected by the
(properly scaled) acceptance criterion. No correctly-implemented metaheuristic
can improve on greedy EFT when a single processor dominates -- greedy is already
optimal in the cost model. The implementation is faithful; the landscape is flat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…h move Two changes from Orsila, Salminen & Hämäläinen (2008), verified against the paper rather than the request sketch: 1. SA_DEFAULT_K 1.0 -> 2.0. §4.3 (Eq. 18/19) states k widens the closed-form temperature range [Tf, T0], with suitable values in [1, 9]; k=1 gives ~27% final-level worsening acceptance, k=2 ~12%. Orsila et al. (2007) use k=2 as their thoroughness/cost tradeoff, so k=2 is the paper's own cited value and is adopted here. (The sketch proposed k=3 to "fix a too-small T0 under the cost cliff"; that premise is backwards -- by Eq. 18 the GPU cliff makes t_min_sum tiny so T0 is already large -- and no in-range k lets SA accept a 250x off-GPU move anyway, so the exact value matters little; k=2 is the defensible cited choice.) 2. ECP (Enhanced Critical Path) move, Orsila §3.7.1 (Wild et al. 2003, reported "significantly better than single move with directed acyclic graphs"). Bias the SA move's task selection toward the critical path -- proxied by task_finish_ns, since the sink has the maximum finish and near-critical tasks finish late. With probability SA_ECP_BIAS_PROB=0.5 select by finish-time roulette (so every on/near-critical-path task is a likely target, the sink most likely), else uniform to keep the neighborhood ergodic. The sketch took the single max-finish task deterministically, which retries the same sink on every biased move and wastes diversity; roulette over finish time favors the whole path, not just its endpoint. The existing move already excludes the current proc (Orsila §3.6), so that correctness detail needed no change. Expected impact is small on these workloads (GPU-dominated placement and hierarchically-partitioned ~17-task sub-DAGs leave little critical-path room), but both changes make SA faithful to the paper. Regression-safe: SA still tracks a separate monotone best and returns it. Existing SA tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…areness) Greedy's data-ready for a `Chunk` argument previously accounted only for moving the tile from its *initial* materialized location (`_greedy_arg_ready_time_ns` on `::Chunk`) -- it had no notion of a producer task, so a consumer looked ready at t=0 even while its input was still being computed. That made greedy (and its `cost_of_schedule` makespan) blind to the DAG's critical path: it optimized a precedence-free bin-packing, not a real schedule. This is the actual flattening factor for the heuristics (the missing `dag_spec.g` edges were not -- greedy never read them; controlled A/B confirmed edges vs no-edges gave identical greedy output). Add a producer-finish term in `_greedy_earliest_data_ready_ns_cached`: when a `last_writer` map is supplied, a `Chunk` consumer waits for the most recent task that wrote that exact Chunk to finish, plus the transfer of its output to the target proc. `greedy_schedule!` builds the map by object identity as it walks tasks in index order (topological), so every producer is registered before its consumers. Chunk size and the cache's per-(proc,proc) move rate give the transfer, consistent with the existing DTask-arg path. Scoped to the standalone GreedyScheduler via `producer_finish=true`; off by default. IG/SA build their seed with the flag off so it stays consistent with their `_replay_schedule!` search (untouched, per plan) -- extending producer-finish to replay + the SA proposer is the follow-up if this moves the Greedy numbers. Measured, cholesky nt=4 CPU+GPU (same DAG + metrics): greedy makespan estimate 18.164ms (off) -> 68.233ms (on); the on value reflects the true critical path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the ROSS 2026 Table-1 benchmark and the two supporting changes it needs;
required to reproduce the grid on other machines (scheduler logic is already on
the branch, but the harness was not).
- bench/datadeps_schedulers/paper_grid.jl: the grid (2 regimes x 2 workloads x
{2,4,8} tiles x {RR,Greedy,IG,SA(IG),JuMP} x 7 seeds). Per-config metrics
warmup, seed-outer scheduler-inner order, per-cell schedule-cache reset
(fresh schedule per seed), records wall/aot/residual/n_tasks/milp_status/
milp_obj. IG/SA/JuMP capped at 60s (PSZ fair-comparison). SA inner is IG.
- ext/JuMPExt.jl: LAST_MILP_SOLVE Ref records the last solve's termination
status and proven-optimal makespan (value(t_last_end), ns) for the paper's
Greedy-vs-MILP optimality-gap number. Behavior-preserving (written after the
existing solve, read by the harness).
- bench/datadeps_schedulers/driver.jl: type datadeps_dag_equivalent's DAGSpec
args on the TimedScheduler override so it is strictly more specific than
Dagger's method -- untyped args are ambiguous and MethodError once a non-empty
schedule cache triggers an equivalence lookup on the hierarchical path.
- bench/datadeps_schedulers/Project.toml: bench env (CUDA/Dagger/HiGHS/JuMP) so
runners can use --project=bench/datadeps_schedulers + Pkg.develop the local
Dagger, instead of the root Project.toml deps hack (which stays uncommitted).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…demo) Self-contained single-session harness the HPC teammate runs once on 4 MI300a APU nodes. Three node regimes (M1/M2/M4) scoped from Dagger.all_processors() grouped by hostname; 2 workloads x 3 tile counts x 5 schedulers x 7 seeds, per-config warmup, seed-major ordering. After the main grid: AOT-only optimality-gap pass on every OPTIMAL MILP cell (cost_of_schedule vs proven makespan), plus a schedule-cache demonstration (two back-to-back AOT calls per regime). Graceful per-cell failure handling, 5-min disk flush, --smoke validation, and a 4-commit tree-state check at startup. Env activates in-script (AMDGPU backend, develop-mode Dagger); launch mechanism left to the teammate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds cross-
spawn_datadepsreuse of hierarchical AOT schedules, so iterative workflows (typical for solvers and stencil codes) pay the AOT scheduling cost once per DAG shape instead of every call. Together with two related bug fixes surfaced along the way, this makes the full scheduler test suite pass onjps/riteshsc26(486 pass / 1 pre-existing broken).The design was pre-discussed on Slack and Julian gave the green light on the "cache the whole hierarchical partitioning" approach.
Cache under hierarchical scheduling
distribute_tasks_hierarchical!now:DAGSpecup front (using the samedag_add_task!logic as the flat path'sdatadeps_build_schedule!) and consultsqueue.scheduler's task-local schedule cache before spinning up any partitioning or per-partition scheduling.DAGSpecScheduleand threads it through the parallel per-partition pipeline via a newprecomputed_schedulekeyword onschedule_partition_full!. Each partition filters the full-region schedule to its own tasks and skips thedatadeps_build_schedule!call entirely; tasks whose cached processor is not inlocal_procs(possible ifall_procschanged since the schedule was cached) fall back to JIT via the existing guard in_schedule_vertex!.DataDepsState, and persists them toqueue.scheduler.cachekeyed by the full-regionDAGSpec.The per-partition
similar(queue.scheduler)shard remains untouched on the miss path (its concurrency and per-partition processor-list reasoning is unchanged); the cache lookup / persistence sits strictly above that layer so no shared mutable state or cross-thread synchronisation is introduced.Bug fixes surfaced by the cache work
1.
_eft_runtime_nsview() on Tuple fargs (commit00e67b0c)_eft_runtime_nsdid@view spec.fargs[2:end], which breaks under typed tasks (DTaskSpec{true}) becausespec.fargsis aTupleandBase.view(::Tuple, ::UnitRange)isn't defined. Dispatch Tuple →Base.tail, Vector →@view(zero allocation on both paths).2. Missing
Base.similarfor parameterised schedulers (commit80df355f)IteratedGreedyScheduler,SimulatedAnnealingScheduler,OptimizingScheduler, andJuMPSchedulerhad noBase.similarmethods, so hierarchical partition-shard creation fell through to the defaultsimilar(::DataDepsScheduler) = typeof(s)()and raised aMethodError(no zero-arg constructor for the parameterised types). Added type-specific methods that preserve configuration and deep-copy the RNG so parallel partition shards produce independent, race-free samples without behavioural drift.3.
JuMPExtMetricsTracker import (commit80df355f)ext/JuMPExt.jlimportedMetricsTrackeras a top-level package, butjps/riteshsc26folds MetricsTracker into theDaggermodule via a directinclude(seesrc/Dagger.jl). Changed toimport Dagger.MetricsTracker as MTso the JuMP extension loads under this branch.Verified
Full
test/datadeps/scheduling.jlonjps/riteshsc26 @ 0c9aff81(Julia 1.12.3, macOS aarch64):Determinism spot check
matmul in
spawn_datadepswith anIteratedGreedyScheduler:Compute-budget rationale
Under the previous behaviour, each
spawn_datadepscall on the same DAG re-ran the full AOT scheduler through hierarchical partitioning. For repeated calls (Felipe's tuolumne re-runs, 3 trials + 1 warmup per cell), this meant 4× the AOT cost per cell. With this PR, only the first call pays; subsequent calls hit the cache. On a matmul K=512 SA cell, that trims ~35 min × 3 = ~1h45m off per (workload, K, node) point.