Skip to content

perf!: make importing Auto3D 44x cheaper, and harden the tests that guard the coming refactors - #138

Merged
isayev merged 4 commits into
mainfrom
debt/wave1-test-hardening-and-import-cost
Aug 4, 2026
Merged

perf!: make importing Auto3D 44x cheaper, and harden the tests that guard the coming refactors#138
isayev merged 4 commits into
mainfrom
debt/wave1-test-hardening-and-import-cost

Conversation

@isayev

@isayev isayev commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Wave 1 of the remaining debt closure: the test hardening that has to land before the
refactors move the code it covers, plus the import-cost fix.

The headline

import Auto3D no longer imports torch or RDKit.

before after
import Auto3D 1.35 s 0.031 s
len(sys.modules) 1175 154 (stdlib floor is 128)
torch / RDKit loaded yes no
Auto3D.* submodules loaded 20 0

Three eager optional-dependency probes in __init__.py defeated the _LAZY_API
mechanism that exists to prevent exactly this. Probing for ANI2xt reached batch_opt,
which reached the utils barrel, which reached validation, which imported torch and
models.loading. Importing the package paid for the entire package before the caller
asked for anything.

The probes are deleted rather than lazified: they are not public API, and every real probe
already exists at its use site. A comment in the file records that and says not to
reintroduce one.

Breaking: Auto3D.ANI2xt, Auto3D.warnings, Auto3D.version and
Auto3D.PackageNotFoundError are gone from the package namespace. None was public API —
all four leaked in via those probes and module-level imports. Documented public names are
unaffected and still resolve lazily. Recorded in the CHANGELOG.

Why the test hardening comes first

A test that asserts nothing cannot tell you whether a refactor preserved behavior, and
hardening it after the refactor moved its code proves nothing about the move. So the
hardening is scoped by a rule rather than by judgment: harden a finding now if and only if
the test it names covers a src/ file that the next refactors touch. Everything else is
deferred to the per-module cleanup sweeps.

Re-verifying the vacuous-test corpus against current source changed the numbers
substantially. The "~105 findings" is 64 distinct finding-groups — that report's Tier 3
list has 24 entries but only 15 distinct, the other 9 being cross-references to items
already counted, and the ~105 counted test functions rather than findings. Of the 64,
26 are in scope for this PR and 38 are deferred. Unlike the dead-code findings, none
had decayed: 31 were re-verified individually and all 31 were still exactly as vacuous as
reported
.

Two of the three tests the audit called missing already exist, and are more thorough than
the literal ask — the force-sign toy-NNP test and the end-to-end pipeline assertions. Only
the narrow gaps were filled.

Tests that were passing for the wrong reason

  • test_embed_single_with_dynamic_conformers used ethane, which is conformationally
    degenerate and always collapses to one conformer, so the assertion held regardless of
    whether dynamic conformer counting worked. Now hexane, bounded against
    calculate_conformer_count.
  • test_chunk_meta_structure built its expected dict by hand from the same literal it then
    compared against. Now derived from ChunkMeta's own annotations and required keys.
  • test_fire_independent_molecule_tracking asserted that per-molecule state existed, not
    that it was independent. Rewritten as a two-phase protocol — bootstrap with every
    molecule progressing, then make exactly one oscillate — asserting the exact a and dt
    values that follow.
  • test_one_bad_molecule_does_not_remove_the_others parsed IDs with a stale
    split("_")[0], which made a Tier 1 test vacuous.
  • Three pytest.raises had no match=: one caught bare Exception, and two could have
    passed on any unrelated ModelLoadError rather than the argument-order check they exist
    for.

Coverage added where a path had none

  • The arity check's len(required) > 3 half was never exercised — only too-few-arguments.
  • The alias vocabulary is now parametrized over the full synonym set in correct and
    transposed order (14 cases), calling validate_custom_nnp directly so nothing pickles.
  • forces = -grad is written separately in ANI2xtAdapter and ANI2xAdapter and was
    tested in neither. Both now assert -2*coords against a toy quadratic model.
  • The "produced no conformers after clash relief" warning had no test at all. One now
    drives the real embedding path with relieve_clash stubbed by atom count, asserting the
    warning fires once, names the right molecule, and that a sibling species survives without
    being warned about.
  • Padding invariance gains an ANI2x case at 1e-3, slow-marked. Not run locally — torchani
    is absent and no NNP may be loaded on that box, so CI verifies it.

Two defects pinned rather than fixed

Both are xfail(strict=True), so they cannot be forgotten and cannot be fixed silently.
Both belong to clusters landing later.

  1. filter_unique_optimized raises KeyError on a record with no E_tot, while the
    legacy filter_unique tolerates it. The two conformer filters were given a shared
    duplicate criterion but still disagree on malformed input — and the stricter one is the
    survivor, so this must be settled before the other is deleted.
  2. opt_steps carries two different minimums. FIELD_BOUNDS declares ("ge", 1)
    while validation.py hand-writes >= 10 in two places. The test asserts the two
    agree rather than picking a value, so whichever is authoritative can win.

Two tests deleted rather than hardened

test_config_schema_exists cannot fail once the module imports, and no field on
Auto3DOptions has a mutable default for test_immutable_default_list to guard.

Slow-tier correction

test_calc_spe_uses_model_factory and test_model_name2model_calculator_uses_factory were
slow-marked despite monkeypatching every model construction. Both move to the fast tier.
Each file's module-level pytestmark became per-test decorators so the 15 genuine real-NNP
tests in those two files stay slow — verified by collecting -m "not slow" and confirming
exactly those two appear.

Verification

  • 1302 passed, 9 skipped, 2 xfailed (fixed order), re-run on the committed tree.
  • Import-cost fix: five tripwires confirmed red first for their stated reasons, then each
    fix mutation-verified by reverting it and confirming the named test goes red — including
    reintroducing the ANI2xt probe (4 red) and adding a cache to __getattr__ (the
    non-caching test red).
  • Cost tests assert module count, not wall-clock seconds, and run in a subprocess since
    the parent pytest process has already imported everything.
  • __getattr__ deliberately does not cache. Caching turns it into a capturing binding,
    which is how a lazily imported reader holds a stub permanently while monkeypatch
    reports success; test_lazy_torchani_import documents that failure surfacing 182 tests
    downstream. A test asserts the non-caching property.
  • Eight-seed order-independence sweep.

Follow-ups this PR does not do

  • get_device, CustomNNP and IsomerEngineFactory are in api.rst but not __all__;
    generate_conformers is the reverse. Reconciling them needs an owner holding both
    api.rst and test_public_api.py.
  • batch_opt/batchopt.py:17-28 has twin dead probes. Harmless now that nothing imports
    batchopt at package-import time, but still dead; goes to the cluster that owns that
    file.

isayev added 4 commits August 4, 2026 00:56
…ions

These cover the code the next several refactors move, so they are hardened
first: a test that asserts nothing cannot tell you whether a refactor
preserved behavior, and hardening it afterwards proves nothing about the
move.

The substantive changes, rather than the mechanical ones:

test_parallel_embed's dynamic-conformer test used ethane, which is
degenerate and always collapses to a single conformer, so the assertion
held for a reason unrelated to what it claimed. It now uses hexane and
bounds the result against calculate_conformer_count.

test_SDF2chunks compared only chunk counts; it now reparses each chunk and
asserts the molecules match the source by name, order and atom count.

test_config's chunk-meta test built its expectation by hand from the same
literal it then compared against. It now derives from ChunkMeta's own
annotations and required keys.

Two tests are deleted rather than hardened. test_config_schema_exists
cannot fail once the module imports, and no field on Auto3DOptions has a
mutable default for test_immutable_default_list to guard.

Two xfail(strict=True) tests pin defects found while deriving the correct
assertions. Both fail today for the reason named in the marker:

- filter_unique_optimized raises KeyError on a record with no E_tot
  property, while the legacy filter_unique tolerates it. The two filters
  were given a shared duplicate criterion but still disagree on malformed
  input, and the stricter one is the survivor.
- opt_steps carries two minimums: FIELD_BOUNDS declares ("ge", 1) while
  validation.py hand-writes >= 10 in two places. The test asserts the two
  agree rather than picking a value, so whichever is authoritative can win.

Verified: 1299 passed, 9 skipped, 2 xfailed.
Three eager optional-dependency probes in __init__.py defeated the _LAZY_API
mechanism that exists to avoid exactly this. Probing for ANI2xt reached
batch_opt, which reached the utils barrel, which reached validation, which
imported torch and models.loading. So `import Auto3D` paid for the entire
package plus torch and rdkit before the caller had asked for anything.

Measured in a subprocess on one box:

  import Auto3D        1.35 s -> 0.031 s
  len(sys.modules)      1175  -> 154      (stdlib floor is 128)
  torch, rdkit loaded    yes  -> no
  Auto3D.* submodules     20  -> 0

The probes are deleted rather than made lazy: they are not public API, and
every real probe already exists at its use site. A comment records that and
says not to reintroduce one.

__version__ now comes from a private _detect_version(), and __dir__ reports
the lazy API, so `main` is discoverable and warnings, version, ANI2xt and
PackageNotFoundError stop leaking into the namespace.

__getattr__ deliberately does not cache. Caching would turn it into a
capturing binding, which is how a lazily imported reader ends up holding a
stub permanently while monkeypatch reports success; test_lazy_torchani_import
documents that failure appearing 182 tests downstream. A test asserts the
non-caching property.

The cost tests assert module count rather than wall-clock seconds, and run in
a subprocess since the parent pytest process has already imported everything.

Verified: five tripwires confirmed red first for the stated reasons, then
each fix mutation-verified by reverting it and confirming the named test goes
red. 1298 passed across three random seeds plus fixed order; ruff clean.
…tributes

Auto3D.ANI2xt, .warnings, .version and .PackageNotFoundError were never
public API, but removing them is still visible to anyone who reached for
them, so they belong in Breaking Changes rather than only in the perf note.
The same hardening as the previous commit, for the tests covering the code
the model-contract consolidation is about to move.

The assertions that were doing no work:

- test_create_unknown_model_raises_error caught bare Exception, so it could
  not tell an unresolvable registry name from any other failure. It now
  matches the message.
- Two pytest.raises(ModelLoadError) had no match=, so either would have
  passed on an unrelated ModelLoadError. Both now match on the argument
  order they exist to check.
- test_ensemble_opt_returns_convergence_info ignored the convergence info it
  is named for; it now asserts the mask and the oscillation counts.
- test_fire_independent_molecule_tracking asserted that per-molecule state
  existed, not that it was independent. Rewritten as a two-phase protocol:
  bootstrap with every molecule progressing, then make exactly one
  oscillate, and assert the exact a and dt values that follow.
- test_optimizer_handles_empty_file accepted any log output; it now asserts
  the empty-file message and that the missing-file message is absent, since
  those two guards were indistinguishable.

New coverage where a code path had none:

- The arity check's len(required) > 3 half was never exercised. Only the
  too-few-arguments half was.
- The alias vocabulary is now parametrized over the full synonym set in both
  correct and transposed order, 14 cases, calling validate_custom_nnp
  directly so nothing needs to pickle.
- forces = -grad is written separately in ANI2xtAdapter and ANI2xAdapter and
  was tested in neither. Both now assert -2*coords against a toy quadratic
  model, hermetically, by bypassing each adapter's real __init__.
- The "produced no conformers after clash relief" warning had no test. One
  drives the real embedding path with relieve_clash stubbed by atom count,
  and asserts the warning fires once, names the right molecule, and that a
  sibling species survives without being warned about.
- test_one_bad_molecule_does_not_remove_the_others parsed IDs with a stale
  split("_")[0] that made a Tier 1 test vacuous.

test_calc_spe_uses_model_factory and test_model_name2model_calculator_uses_factory
were slow-marked despite monkeypatching every model construction. Both move to
the fast tier; the module-level pytestmark becomes per-test decorators so the
15 real-NNP tests in those two files stay slow. Verified: exactly those two
are collected by -m "not slow".

Padding invariance gains an ANI2x case at 1e-3, slow-marked. It is not run
here: torchani is absent locally and no NNP may be loaded on this box, so CI
is what verifies it.

Verified: 1302 passed, 9 skipped, 2 xfailed.
@isayev
isayev merged commit 97a64b4 into main Aug 4, 2026
8 checks passed
@isayev
isayev deleted the debt/wave1-test-hardening-and-import-cost branch August 4, 2026 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant