Add populate_cache for batched multi-RHS warming of virtual matrices#314
Open
jd-lara wants to merge 6 commits into
Open
Add populate_cache for batched multi-RHS warming of virtual matrices#314jd-lara wants to merge 6 commits into
jd-lara wants to merge 6 commits into
Conversation
Introduce `populate_cache` for VirtualPTDF, VirtualLODF, and VirtualMODF so callers can pre-warm row caches over an iterable of components/contingencies using the solver backends' batched multi-RHS `solve_sparse!` instead of one single-RHS solve per row. This amortizes the ABA factorization solves that dominate optimization-problem builds on large systems. - VirtualPTDF / VirtualLODF: gather requested arc rows, build one sparse RHS of the corresponding BA columns, solve in a single batched call, then scatter (PTDF) or apply the LODF map (A*X .* inv_PTDF_A_diag) across all columns at once. Accepts integer indices, arc bus-pair tuples, or branch-name strings. - VirtualMODF: batch the pre-contingency BA-column solves over the union of all monitored arcs and every contingency's modified arcs. The monitored-arc solve is contingency-independent, so it is computed once and reused; Woodbury factors per contingency are assembled from the precomputed solves with no further linear solves. Monitored set is supplied by the user. - RowCache: add set_persistent_row!/pin_row! so populated rows are pinned and never evicted by later lazy lookups, plus warn_if_over_capacity. - Backend dispatch routes to KLU's solve_sparse! or AccelerateWrapper's, with a column-by-column fallback for other factorization backends. - Export populate_cache; add tests covering PTDF/LODF/MODF equivalence with the lazy getindex path, pinning, identifier resolution, and error handling. https://claude.ai/code/session_011TFwa4Hsdse4XYEeuHmYro
Contributor
Performance ResultsPrecompile Time
Execution Time
|
…property Merge of origin/psy6 moved the virtual matrices' shared state into VirtualFactorCore. Rework populate_cache so every wrapper field is read through its proper getter (get_core, get_cache, get_cache_lock, get_dist_slack, get_dist_slack_normalized, get_inv_PTDF_A_diag, get_row_caches, get_woodbury_cache, get_max_cache_size_bytes, get_contingency_cache, get_arc_lookup, get_tol, get_ref_bus_position) instead of relying on the wrappers' getproperty forwarding. Core container fields are accessed directly on the VirtualFactorCore, matching _compute_ptdf_row/_compute_lodf_row and the Woodbury kernel. The batched multi-RHS solve and the from-precomputed-solve Woodbury assembly are unchanged and remain byte-for-byte consistent with the post-merge woodbury_kernel.jl. Tests read cache/core/contingency state through getters too.
The redesigned backend routes array getindex through to_index, which resolves the row against the arc-tuple lookup, so vptdf[i, :] with an integer row no longer works (integer+integer and arc-tuple+colon do). Rework the PTDF/LODF tests to index rows by arc tuple, and cover integer-keyed population by inspecting the row cache directly. VirtualMODF keeps its integer getindex(::Int, contingency) method, so those tests are unchanged in spirit. Also apply the JuliaFormatter wrapping flagged by reviewdog on src/populate_cache.jl (generic _solve_multi_rhs!, the mods comprehension, and the _woodbury_factors_from_base call) and restructure the solver-looped testsets to a begin/for form that stays within the line-length margin.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new public API populate_cache to bulk pre-warm and pin lazy row caches for VirtualPTDF, VirtualLODF, and VirtualMODF using the existing batched multi-RHS solve_sparse! backends, reducing “one-solve-per-row” overhead during large optimization-model builds.
Changes:
- Introduces
populate_cacheimplementations for VirtualPTDF/LODF/MODF, including batched BA-column solves and MODF Woodbury assembly from precomputed base solves. - Extends
RowCachewith pinned-row helpers (set_persistent_row!,pin_row!) and a capacity warning (warn_if_over_capacity). - Adds a dedicated test file covering equivalence vs lazy
getindex, pinning semantics, identifier resolution, and key error paths.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
src/populate_cache.jl |
New populate_cache API + batched solve plumbing and MODF batched warm path. |
src/row_cache.jl |
Adds APIs to pin rows persistently and warn when pinned rows exceed configured capacity. |
src/PowerNetworkMatrices.jl |
Exports populate_cache and includes the new implementation file. |
test/test_populate_cache.jl |
Adds coverage for populate_cache behavior across PTDF/LODF/MODF and backends. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+33
to
+45
| # Generic fallback for any other factorization backend: solve column-by-column. | ||
| function _solve_multi_rhs!( | ||
| K, | ||
| B::SparseArrays.SparseMatrixCSC{Float64, Int}, | ||
| out::Matrix{Float64}, | ||
| ) | ||
| n = size(B, 1) | ||
| col = zeros(n) | ||
| @inbounds for j in axes(B, 2) | ||
| fill!(col, 0.0) | ||
| for p in SparseArrays.nzrange(B, j) | ||
| col[SparseArrays.rowvals(B)[p]] = SparseArrays.nonzeros(B)[p] | ||
| end |
Comment on lines
+67
to
+70
| # --- Component resolution -------------------------------------------------- | ||
|
|
||
| # Resolve a user-supplied component to an integer arc/row index. Accepts the | ||
| # same identifiers as `getindex`: an integer index, an arc bus-pair tuple, or a |
Comment on lines
+113
to
+127
| if !use_dist_slack && !isempty(dist_slack) | ||
| error("Distributed bus specification doesn't match the number of buses.") | ||
| end | ||
| tol = get_tol(core) | ||
|
|
||
| # Only solve rows not already resident; existing rows are pinned below. | ||
| new_rows = @lock cache_lock Int[r for r in rows if !haskey(cache, r)] | ||
|
|
||
| if !isempty(new_rows) | ||
| sol = @lock core.solver_lock _solve_arc_columns(core, new_rows) | ||
| valid_ix = core.valid_ix | ||
| @lock cache_lock begin | ||
| full = zeros(buscount) | ||
| for (j, r) in enumerate(new_rows) | ||
| haskey(cache, r) && continue # lost a race; keep the winner |
…back - VirtualMODF populate_cache resolves monitored arc tuples through _monitored_arc_index, so an arc eliminated by network reduction yields the descriptive VirtualMODF error instead of a raw KeyError. - VirtualPTDF/VirtualLODF populate_cache build each stored row (scatter, dist-slack, sparsify) outside cache_lock and take the lock only for the double-check + insert, matching the cached_row_lookup pattern and shortening the critical section. - The generic _solve_multi_rhs! fallback now checks `applicable` and raises a clear, actionable error when a backend implements neither solve_sparse! nor _solve_factorization, instead of surfacing a raw MethodError. - Add shared RowCacheValue alias for the cached-row value type.
Replace the remaining `vmodf.arc_susceptances` field reads in the populate_cache tests with the `_get_arc_susceptances` accessor, so no code in this change reaches struct fields through the wrappers' getproperty forwarding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new exported function
populate_cacheforVirtualPTDF,VirtualLODF, andVirtualMODFthat lets callers pre-warm the lazy row caches over an iterable of components/contingencies using the solver backends' batched multi-RHSsolve_sparse!, instead of the one-single-RHS-solve-per-row thatgetindexperforms today.This is targeted at
psy6for integration into the re-designed back end. The goal is to accelerate the VirtualPTDF and VirtualMODF queries that dominate the build of large optimization problems.Motivation
getindexon the virtual matrices computes one row at a time, issuing a single right-hand-side solve through the ABA factorization per row. When an optimization-problem build queries many rows (every monitored branch, every contingency), those solves dominate runtime. Both solver backends (KLULinSolveCache,AAFactorCache) already expose a batchedsolve_sparse!(K, B, out; block=64)primitive — used today only by the fullPTDF/LODFbuilders.populate_cachereuses that primitive on the virtual path.What changed
BA[valid_ix, rows]columns, solve in a single batched call, then scatter back to bus space (PTDF) or apply the LODF map(A·X) .* inv_PTDF_A_diagacross all columns at once. Accepts integer arc indices, arc bus-pair tuples(from, to), or branch-name strings. The per-row math is identical to the existing single-row_compute_*_row.B⁻¹·BA[:,arc]solves over the union of every contingency's modified arcs and the user-supplied monitored set. Because the pre-contingency monitored-arc solve is contingency-independent, it is computed once and reused; the per-contingency Woodbury factors are assembled from these precomputed solves with no further linear solves. This collapsesO(n_contingencies × (M + n_monitored))one-at-a-time solves into a single batched solve over the distinct arcs. The monitored set is supplied by the user (monitoredkeyword).set_persistent_row!/pin_row!so populated rows are pinned (added topersistent_cache_keys) and never evicted by later lazy lookups, pluswarn_if_over_capacityto flag undersized caches.solve_sparse!orAccelerateWrapper.solve_sparse!, with a column-by-column fallback for any other factorization backend.populate_cache; the public API docs page picks it up automatically via@autodocs.API
Tests
test/test_populate_cache.jlcovers, for KLU and (when available) AppleAccelerate:populate_cachepath and the lazygetindexpath,Note
Julia was not available in the execution environment, so the suite was not run locally — correctness was validated by close review against the existing solve/cache code paths and Aqua constraints (exports, ambiguities, unbound args). CI on this PR will exercise it.
https://claude.ai/code/session_011TFwa4Hsdse4XYEeuHmYro
Generated by Claude Code