Conversation
timtreis
force-pushed
the
feat/tvn-coral
branch
from
September 20, 2026 21:13
ae38dbe to
fa9cfda
Compare
pp.tvn is typical variation normalization as EFAAR defines it: centre and scale on the controls, rotate onto the principal components of the controls alone, centre and scale on them again within each batch, then CORAL each batch by whitening with the covariance of its own controls and recolouring with the covariance of all of them. sphere already whitens by the control covariance, but its per-group fit cannot express the part that matters here, a target taken from every batch at once. The rotation is fitted on the controls and so keeps one component per control at most, which is why this writes obsm and never X. Matched against EFAAR_benchmarking at 2935f21 in the tests, including the population standard deviation of the centre-scale against the sample covariance of CORAL. The matrix roots go through eigh rather than a general fractional matrix power: the regularized covariances are symmetric, so the decomposition is exact and real. On control-normalized CellProfiler features it did not beat plain principal components on either screen this package ships. The docstring says so, and says that the published gains are on learned embeddings.
The first measurement used BBBC021 and the eleven-source Target-2 plates, both CellProfiler features, and read as TVN losing outright. Measured again on JUMP-Lite, four plates from four laboratories with 64 control wells each, it roughly doubles recall of compounds sharing an annotated target on all six feature sets, the CellProfiler-equivalent one included, while replicate retrieval falls. The trade is between those two, not between feature types, and what it needs is controls per batch.
…ffsets A dimension with no spread among a batch's controls was silently clamped to a scale of 1. pp.normalize warns about the same condition and flags var; here var cannot be flagged, because the columns being scaled are an embedding it does not describe, so the warning names the batch instead. The per-batch pass is where this bites, being fitted on that batch's controls alone. Both passes over the batches now take their rows from group_offsets, which _core/_reduce.py documents as the rule and nineteen call sites follow, rather than scanning codes == group once per batch. The astype(float64) went: representation already returns float64, so it only ever copied the matrix. use_rep defaults to X_pca, as pp.harmony's does, since this is a batch correction on an embedding; None still reads X.
timtreis
force-pushed
the
feat/tvn-coral
branch
from
September 20, 2026 21:49
ae04f4f to
a4538bc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #99, which it is measured with.
pp.tvnis typical variation normalization as EFAAR defines it: centre and scale on the controls, rotate onto the principal components of the controls alone, centre and scale on them again within each batch, then CORAL — whiten each batch by the covariance of its own controls and recolour it with the covariance of all of them.pp.spherealready whitens by a control covariance, but its per-group fit cannot express a target taken from every batch at once.The rotation is fitted on the controls, so it keeps at most one component per control and the result can be narrower than the input. That is why this writes
obsmand neverX.Checked against
EFAAR_benchmarkingat 2935f21 by transcribing the reference into the test, which pins the order of operations and the two different degrees of freedom it uses. The matrix roots go througheighinstead of a general fractional matrix power, because the regularized covariances are symmetric, so the decomposition is exact and real rather than complex with a discarded imaginary part.What it does, measured. It trades replicate consistency for relationship recall. Across the six feature sets of
ds.jump_lite— five learned embeddings and one CellProfiler-equivalent, four plates run at four laboratories — recall of compounds sharing an annotated target rose sharply on every one, and replicate retrieval fell on all but one. It needs controls per batch, not in total, since each batch is whitened by its own controls' covariance: on BBBC021, whose batches are 55 plates with six control wells each, it lowered mechanism retrieval at every width tried, while 64 control wells per batch was enough. The docstring says this, and says the choice between those two readouts is the question to answer first.An earlier revision of this PR said it lost outright. That was measured only on CellProfiler features, and where annotations were missing only on replicate retrieval, so it was too strong.
Verified: 805 tests pass, mypy, ruff and the formatters clean.