Conversation
parafac2 touches its input matrix through matmul/rmatmul (already supported via __matmul__/__rmatmul__) plus a squared-Frobenius-norm reduction it currently only knows how to compute for a plain np.ndarray or scipy.sparse array. Add that as norm_sq()/slice_norms() on NormalizedViewBase, computed with new numba kernels that reuse the same O(nnz) traversal as the existing matmul/toarray kernels, so a normalized view can satisfy parafac2's duck-typed backend contract without materializing anything. Also add to_scipy_sparse() (the uncentered, scaled sparse term with the same sparsity pattern as the raw array) and a means property (the per-gene correction to subtract from it), for code that only knows how to move a plain NumPy/SciPy array onto a device -- e.g. so a normalized view can be materialized into a real (and, for typical single-cell data, tiny) sparse array before running through parafac2's existing CuPy/MLX GPU path, rather than needing new GPU-native kernels of its own. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
VCSCAnnData stores X/raw_X in private _vcs_X/_vcs_raw_X attributes (anndata's own X validation rejects a VCSCArray/VCSRArray directly), but never overrode copy(), so the inherited anndata.AnnData.copy() copies the standard (unused, always-None) _X attribute instead: X silently comes back None, and the returned object is downgraded to a plain AnnData rather than preserving this class (or a subclass, such as one overriding the X property for a lazy-normalized view). Any caller relying on `adata[mask].copy()` -- e.g. parafac2's BiCV train/test splitting -- hits this immediately. Add an explicit override that copies every field, including a real VCSCArray/VCSRArray copy of X/raw_X, and returns type(self). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The inherited anndata.AnnData.to_memory() has the same root cause as copy() (previous commit): it only knows about the standard, unused _X attribute, not this class's _vcs_X/_vcs_raw_X, and reconstructs a plain AnnData that silently loses X. This class never actually supports a lazily backed X/raw_X, so to_memory() now just delegates to copy(). scrise's BiCV rank selection calls to_memory() on its input before the train/test split loop, so this was hit immediately after fixing copy(). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
__getitem__ already returned a new, eagerly-copied object (rather than a lazy view) since anndata's own view machinery can't handle _vcs_X, but it hardcoded the returned type to VCSCAnnData instead of type(self) -- so a subclass overriding X (e.g. one that always hands back a value normalized fresh from _vcs_X, as with copy()/to_memory() in the previous two commits) silently reverted to the raw, un-normalized array after any slice. scrise's BiCV rank selection slices its input for every train/test split, so this surfaced immediately after fixing copy()/to_memory(): the sliced object's X was the raw VCSRArray, which doesn't implement the norm_sq() a normalized view needs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rnel The misaligned direction of a normalized view's matmul (self@B for VCSC, B@self for VCSR) previously regrouped the array into the other VCS format a chunk of major slices at a time (_chunk_bounds/_transpose_major), caching a full opposite-format copy (_dual_arr) when the whole array fit one chunk's budget. At scale (~2.3B nonzeros, 1.3M x 12.2K), this needed ~1,150 chunks per pass, repeated across every power iteration and every rank/trial in a BiCV sweep -- tens of thousands of variably-sized alloc/free cycles that fragmented the allocator and drove RSS to ~140 GB on a shared machine, even though no single chunk's live memory was large. Replace it with a direct minor-axis kernel that walks the array's own storage as-is: each thread gets a private, full-output-sized accumulator and scatters into it while owning a disjoint range of major slices, summed across threads at the end -- the same pattern _ops.py already uses for minor_sums/minor_counts/minor_extrema. accumulator_threads caps the thread count to a fixed byte budget, so the accumulator block is always nthreads * output_dim * width * 8 bytes: bounded, independent of nnz, and allocated once per call instead of thousands of times. For a huge output axis (e.g. millions of cells) that caps down to a single thread, trading parallelism for a hard memory bound rather than the previous unbounded chunk churn. Removes _chunk_bounds/_aligned_source/_build_dual/_dual_arr and the chunked-transpose test suite; adds tests/test_minor_axis_matmul.py covering correctness against a dense reference, thread-count invariance, that no second copy of the array is ever built, and that peak memory stays bounded by the accumulator budget rather than nnz. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tion select() previously always recomputed statistics fresh for the selected sub-array. A caller that instead wants to evaluate a fixed (e.g. train-derived) normalization against a different slice -- such as bi-cross-validation scoring a held-out test block against train-derived statistics -- had no lazy option: bracket indexing (__getitem__) keeps the parent's statistics but is documented to always eagerly materialize the selection as a dense ndarray, which is fine for a small window but not for a selection covering a large fraction of a huge array. select(rows, cols, recalculate=False) reuses this view's existing a/b/c/s (row_scale sliced by rows -- exact, since it's a per-cell quantity; the per-gene stats sliced by cols, unchanged by which rows are selected) via the existing from_stats() classmethod, and returns a real view rather than an array -- so it composes with `@`/toarray() and stays lazy regardless of selection size, unlike __getitem__. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.