perf!: make importing Auto3D 44x cheaper, and harden the tests that guard the coming refactors - #138
Merged
Merged
Conversation
…ions
These cover the code the next several refactors move, so they are hardened
first: a test that asserts nothing cannot tell you whether a refactor
preserved behavior, and hardening it afterwards proves nothing about the
move.
The substantive changes, rather than the mechanical ones:
test_parallel_embed's dynamic-conformer test used ethane, which is
degenerate and always collapses to a single conformer, so the assertion
held for a reason unrelated to what it claimed. It now uses hexane and
bounds the result against calculate_conformer_count.
test_SDF2chunks compared only chunk counts; it now reparses each chunk and
asserts the molecules match the source by name, order and atom count.
test_config's chunk-meta test built its expectation by hand from the same
literal it then compared against. It now derives from ChunkMeta's own
annotations and required keys.
Two tests are deleted rather than hardened. test_config_schema_exists
cannot fail once the module imports, and no field on Auto3DOptions has a
mutable default for test_immutable_default_list to guard.
Two xfail(strict=True) tests pin defects found while deriving the correct
assertions. Both fail today for the reason named in the marker:
- filter_unique_optimized raises KeyError on a record with no E_tot
property, while the legacy filter_unique tolerates it. The two filters
were given a shared duplicate criterion but still disagree on malformed
input, and the stricter one is the survivor.
- opt_steps carries two minimums: FIELD_BOUNDS declares ("ge", 1) while
validation.py hand-writes >= 10 in two places. The test asserts the two
agree rather than picking a value, so whichever is authoritative can win.
Verified: 1299 passed, 9 skipped, 2 xfailed.
Three eager optional-dependency probes in __init__.py defeated the _LAZY_API mechanism that exists to avoid exactly this. Probing for ANI2xt reached batch_opt, which reached the utils barrel, which reached validation, which imported torch and models.loading. So `import Auto3D` paid for the entire package plus torch and rdkit before the caller had asked for anything. Measured in a subprocess on one box: import Auto3D 1.35 s -> 0.031 s len(sys.modules) 1175 -> 154 (stdlib floor is 128) torch, rdkit loaded yes -> no Auto3D.* submodules 20 -> 0 The probes are deleted rather than made lazy: they are not public API, and every real probe already exists at its use site. A comment records that and says not to reintroduce one. __version__ now comes from a private _detect_version(), and __dir__ reports the lazy API, so `main` is discoverable and warnings, version, ANI2xt and PackageNotFoundError stop leaking into the namespace. __getattr__ deliberately does not cache. Caching would turn it into a capturing binding, which is how a lazily imported reader ends up holding a stub permanently while monkeypatch reports success; test_lazy_torchani_import documents that failure appearing 182 tests downstream. A test asserts the non-caching property. The cost tests assert module count rather than wall-clock seconds, and run in a subprocess since the parent pytest process has already imported everything. Verified: five tripwires confirmed red first for the stated reasons, then each fix mutation-verified by reverting it and confirming the named test goes red. 1298 passed across three random seeds plus fixed order; ruff clean.
…tributes Auto3D.ANI2xt, .warnings, .version and .PackageNotFoundError were never public API, but removing them is still visible to anyone who reached for them, so they belong in Breaking Changes rather than only in the perf note.
The same hardening as the previous commit, for the tests covering the code
the model-contract consolidation is about to move.
The assertions that were doing no work:
- test_create_unknown_model_raises_error caught bare Exception, so it could
not tell an unresolvable registry name from any other failure. It now
matches the message.
- Two pytest.raises(ModelLoadError) had no match=, so either would have
passed on an unrelated ModelLoadError. Both now match on the argument
order they exist to check.
- test_ensemble_opt_returns_convergence_info ignored the convergence info it
is named for; it now asserts the mask and the oscillation counts.
- test_fire_independent_molecule_tracking asserted that per-molecule state
existed, not that it was independent. Rewritten as a two-phase protocol:
bootstrap with every molecule progressing, then make exactly one
oscillate, and assert the exact a and dt values that follow.
- test_optimizer_handles_empty_file accepted any log output; it now asserts
the empty-file message and that the missing-file message is absent, since
those two guards were indistinguishable.
New coverage where a code path had none:
- The arity check's len(required) > 3 half was never exercised. Only the
too-few-arguments half was.
- The alias vocabulary is now parametrized over the full synonym set in both
correct and transposed order, 14 cases, calling validate_custom_nnp
directly so nothing needs to pickle.
- forces = -grad is written separately in ANI2xtAdapter and ANI2xAdapter and
was tested in neither. Both now assert -2*coords against a toy quadratic
model, hermetically, by bypassing each adapter's real __init__.
- The "produced no conformers after clash relief" warning had no test. One
drives the real embedding path with relieve_clash stubbed by atom count,
and asserts the warning fires once, names the right molecule, and that a
sibling species survives without being warned about.
- test_one_bad_molecule_does_not_remove_the_others parsed IDs with a stale
split("_")[0] that made a Tier 1 test vacuous.
test_calc_spe_uses_model_factory and test_model_name2model_calculator_uses_factory
were slow-marked despite monkeypatching every model construction. Both move to
the fast tier; the module-level pytestmark becomes per-test decorators so the
15 real-NNP tests in those two files stay slow. Verified: exactly those two
are collected by -m "not slow".
Padding invariance gains an ANI2x case at 1e-3, slow-marked. It is not run
here: torchani is absent locally and no NNP may be loaded on this box, so CI
is what verifies it.
Verified: 1302 passed, 9 skipped, 2 xfailed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wave 1 of the remaining debt closure: the test hardening that has to land before the
refactors move the code it covers, plus the import-cost fix.
The headline
import Auto3Dno longer imports torch or RDKit.import Auto3Dlen(sys.modules)Auto3D.*submodules loadedThree eager optional-dependency probes in
__init__.pydefeated the_LAZY_APImechanism that exists to prevent exactly this. Probing for ANI2xt reached
batch_opt,which reached the
utilsbarrel, which reachedvalidation, which imported torch andmodels.loading. Importing the package paid for the entire package before the callerasked for anything.
The probes are deleted rather than lazified: they are not public API, and every real probe
already exists at its use site. A comment in the file records that and says not to
reintroduce one.
Breaking:
Auto3D.ANI2xt,Auto3D.warnings,Auto3D.versionandAuto3D.PackageNotFoundErrorare gone from the package namespace. None was public API —all four leaked in via those probes and module-level imports. Documented public names are
unaffected and still resolve lazily. Recorded in the CHANGELOG.
Why the test hardening comes first
A test that asserts nothing cannot tell you whether a refactor preserved behavior, and
hardening it after the refactor moved its code proves nothing about the move. So the
hardening is scoped by a rule rather than by judgment: harden a finding now if and only if
the test it names covers a
src/file that the next refactors touch. Everything else isdeferred to the per-module cleanup sweeps.
Re-verifying the vacuous-test corpus against current source changed the numbers
substantially. The "~105 findings" is 64 distinct finding-groups — that report's Tier 3
list has 24 entries but only 15 distinct, the other 9 being cross-references to items
already counted, and the ~105 counted test functions rather than findings. Of the 64,
26 are in scope for this PR and 38 are deferred. Unlike the dead-code findings, none
had decayed: 31 were re-verified individually and all 31 were still exactly as vacuous as
reported.
Two of the three tests the audit called missing already exist, and are more thorough than
the literal ask — the force-sign toy-NNP test and the end-to-end pipeline assertions. Only
the narrow gaps were filled.
Tests that were passing for the wrong reason
test_embed_single_with_dynamic_conformersused ethane, which is conformationallydegenerate and always collapses to one conformer, so the assertion held regardless of
whether dynamic conformer counting worked. Now hexane, bounded against
calculate_conformer_count.test_chunk_meta_structurebuilt its expected dict by hand from the same literal it thencompared against. Now derived from
ChunkMeta's own annotations and required keys.test_fire_independent_molecule_trackingasserted that per-molecule state existed, notthat it was independent. Rewritten as a two-phase protocol — bootstrap with every
molecule progressing, then make exactly one oscillate — asserting the exact
aanddtvalues that follow.
test_one_bad_molecule_does_not_remove_the_othersparsed IDs with a stalesplit("_")[0], which made a Tier 1 test vacuous.pytest.raiseshad nomatch=: one caught bareException, and two could havepassed on any unrelated
ModelLoadErrorrather than the argument-order check they existfor.
Coverage added where a path had none
len(required) > 3half was never exercised — only too-few-arguments.transposed order (14 cases), calling
validate_custom_nnpdirectly so nothing pickles.forces = -gradis written separately inANI2xtAdapterandANI2xAdapterand wastested in neither. Both now assert
-2*coordsagainst a toy quadratic model.drives the real embedding path with
relieve_clashstubbed by atom count, asserting thewarning fires once, names the right molecule, and that a sibling species survives without
being warned about.
is absent and no NNP may be loaded on that box, so CI verifies it.
Two defects pinned rather than fixed
Both are
xfail(strict=True), so they cannot be forgotten and cannot be fixed silently.Both belong to clusters landing later.
filter_unique_optimizedraisesKeyErroron a record with noE_tot, while thelegacy
filter_uniquetolerates it. The two conformer filters were given a sharedduplicate criterion but still disagree on malformed input — and the stricter one is the
survivor, so this must be settled before the other is deleted.
opt_stepscarries two different minimums.FIELD_BOUNDSdeclares("ge", 1)while
validation.pyhand-writes>= 10in two places. The test asserts the twoagree rather than picking a value, so whichever is authoritative can win.
Two tests deleted rather than hardened
test_config_schema_existscannot fail once the module imports, and no field onAuto3DOptionshas a mutable default fortest_immutable_default_listto guard.Slow-tier correction
test_calc_spe_uses_model_factoryandtest_model_name2model_calculator_uses_factorywereslow-marked despite monkeypatching every model construction. Both move to the fast tier.
Each file's module-level
pytestmarkbecame per-test decorators so the 15 genuine real-NNPtests in those two files stay slow — verified by collecting
-m "not slow"and confirmingexactly those two appear.
Verification
fix mutation-verified by reverting it and confirming the named test goes red — including
reintroducing the
ANI2xtprobe (4 red) and adding a cache to__getattr__(thenon-caching test red).
the parent pytest process has already imported everything.
__getattr__deliberately does not cache. Caching turns it into a capturing binding,which is how a lazily imported reader holds a stub permanently while
monkeypatchreports success;
test_lazy_torchani_importdocuments that failure surfacing 182 testsdownstream. A test asserts the non-caching property.
Follow-ups this PR does not do
get_device,CustomNNPandIsomerEngineFactoryare inapi.rstbut not__all__;generate_conformersis the reverse. Reconciling them needs an owner holding bothapi.rstandtest_public_api.py.batch_opt/batchopt.py:17-28has twin dead probes. Harmless now that nothing importsbatchoptat package-import time, but still dead; goes to the cluster that owns thatfile.