Skip to content

refactor!: delete the utils barrel and make utils a leaf package - #140

Merged
isayev merged 1 commit into
mainfrom
debt/wave3-barrel-demolition
Aug 4, 2026
Merged

refactor!: delete the utils barrel and make utils a leaf package#140
isayev merged 1 commit into
mainfrom
debt/wave3-barrel-demolition

Conversation

@isayev

@isayev isayev commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Wave 3: the utils barrel is gone and utils/ is a leaf package. Three findings that had
to move together, because all three are about which module a name is imported from.

utils/__init__.py re-exported 41 names

The recorded cause — "names missing from __all__" — was stale. The file's imports and
its __all__ were an exact bijection.

The real defect is inconsistency. Five of eight submodules were re-exported and three were
not, so energy, convergence and stereo_check had no barrel presence at all. Meanwhile
check_connectivity was reached through the barrel in filtering.py:13 and through its own
module in ranking.py:12 — two sibling files, two different paths to the same function.

utils/__init__.py is now docstring-only, documenting what each of the eight modules owns.
The 14 barrel import lines across 12 files name the defining module instead. An AST test,
scoped to src/Auto3D/**, asserts no first-party module imports through it again — AST
rather than regex, so it cannot be fooled by a mention in a comment or string.

Full 41-name mapping table (old barrel path → new dotted path) is in the CHANGELOG.

batchopt.py's compat barrel was narrower than recorded

Only print_stats was a pure re-export; it is deleted. EnForce_ANI and n_steps are
genuinely used in-module and stay imported, now with a comment saying they are not
re-exports so the next reader doesn't delete them.

Its consumers were exactly two: SPE.py:9 and ASE/thermo.py:26, both now pointing at
batch_opt.model_wrapper. The finding also named ASE/geometry.py, auto3D.py and
workflow_workers.py — those are wrong. They import optimizing, which is defined
there, and are untouched.

utils/ is now a leaf

Both Auto3D.models.* imports in validation.py move to function scope. It was the only
non-leaf module in the package. A subprocess tripwire asserts that importing
Auto3D.utils.validation loads no Auto3D.models module — it was red for the stated reason,
pulling in six of them.

import torch in validation.py stays at module level, deliberately

22 test sites patch Auto3D.utils.validation.torch.cuda.is_available. Deferring that import
breaks all 22 at once — confirmed by performing the mutation and counting, not by
reasoning about it. A test now pins the import scope so the next person doesn't rediscover
this the expensive way. M43 was about the models/ import, not about torch; conflating the
two is the cheap-looking shortcut to an import-cost number that this test now blocks.

The public-surface rule, corrected

The rule proposed for this phase was "public iff in Auto3D.__all__ and in api.rst".
That is wrong, and implementing it literally would have been a mistake: api.rst documents
24 entries, ten of which are Auto3D.exceptions.*, none of which belong at top level.
api.rst documents dotted paths; __all__ is the top-level barrel. They are not the same
set and never were.

So the implication is enforced one way only — exported ⟹ documented, plus every
documented path must resolve. Both directions are enforced by test, and api.rst now states
the rule in prose. __all__ is unchanged at 13 names.

Under that rule, the four names that were inconsistent resolve as:

name decision why
get_device stays api.rst-only documented and referenced as Auto3D.model_factory.get_device (5× in the migration guide). Its apparent notebook uses are torch.cuda.get_device_*, a different thing. A top-level alias would mint a second supported path for nothing.
CustomNNP stays api.rst-only the migration guide names from Auto3D.models import CustomNNP as the removed form and Auto3D.models.contract.CustomNNP as the survivor. A third path would contradict it.
IsomerEngineFactory stays api.rst-only the one reserved package-path exception, which isomers/__init__.py's docstring already teaches.
generate_conformers added to api.rst it has no defining module — it exists only as the top-level alias __init__.py calls canonical. Deleting the intended-canonical name to satisfy a docs list is the wrong direction.

Found while implementing, and not swept under the rug

Two more package barrels violate the rule: cli/__init__.py (6 names) and
models/__init__.py (7 names). Neither is documented at its package path. Both are out of
scope here, so test_only_documented_subpackages_define_all freezes the allowed set at
{isomers, cli, models} with a comment recording the latter two as debt rather than
blessing them. Removing one means deleting it from that frozen set, and nobody can add a
fourth by accident.

Three comments this change falsified, corrected minimally — mechanism updated, conclusion
kept. Two lazy imports (models/species.py, models/preflight.py) were justified by
"import Auto3D.utils reaches this module", which is no longer true, as was a test docstring.
Left as-is they would have invited someone to un-defer rdkit and requests and quietly
undo wave 1's import-cost win.

Verification

  • 1357 passed, 9 skipped, 1 xfailed, ruff clean. (+11 boundary tests, −3 barrel-pinning tests.)
  • The remaining xfail is the E_tot filter divergence, still owned by a later cluster.
  • Four seeds — 12345, 1351916419, 20260804, 7 — all 1357.
  • The documented order-sensitive trio run in order
    (test_lazy_torchani_importtest_torch_configtest_thermo_helpers): 111 passed.
    This is the change most likely to break module identity, since it rewrites import sources
    across 12 modules.
  • All eight tripwires confirmed red first for their stated reasons; all eight mutations
    verified red and restored under sha256 check.

Three findings that had to move together, since all three are about which
module a name is imported from.

utils/__init__.py re-exported 41 names. The recorded cause -- names missing
from __all__ -- was stale: the file's imports and its __all__ were an exact
bijection. The real defect is inconsistency. Five of eight submodules were
re-exported and three were not, so energy, convergence and stereo_check had no
barrel presence at all, while check_connectivity was reached through the barrel
in filtering.py and through its own module in ranking.py, one file apart.

The barrel is now docstring-only, documenting what each of the eight modules
owns, and the 14 import lines across 12 files name the defining module. An AST
test, scoped to src/Auto3D and not regex-based, asserts no first-party module
imports through it again.

batchopt.py's compat barrel was narrower than recorded: only print_stats was a
pure re-export, and it is deleted. EnForce_ANI and n_steps are used in-module
and stay, with a comment saying why they are not re-exports. Its consumers were
exactly SPE.py and ASE/thermo.py; the three other sites in the finding import
optimizing, which is defined there, and are untouched.

utils/ is now a leaf. Both Auto3D.models imports in validation.py move to
function scope, verified by a subprocess test asserting that importing
Auto3D.utils.validation loads no Auto3D.models module.

validation.py's module-level `import torch` is deliberately left where it is.
22 test sites patch Auto3D.utils.validation.torch.cuda.is_available, so
deferring it breaks all 22 at once -- confirmed by actually deferring it and
counting. A test pins the import scope so a future reader cannot discover this
the expensive way.

On the public surface: api.rst documents dotted paths, __all__ is the
top-level barrel, and these are not the same set -- ten of api.rst's entries
are Auto3D.exceptions.*, which do not belong at top level. So the rule
enforced is one-directional: exported implies documented, and every documented
path must resolve. Under it get_device, CustomNNP and IsomerEngineFactory stay
documented at their own dotted paths rather than gaining top-level aliases,
since a second supported path buys nothing and, for CustomNNP, would contradict
the migration guide that names Auto3D.models.contract.CustomNNP as the
survivor. generate_conformers is documented instead of removed: it has no
defining module, existing only as the alias __init__.py calls canonical.

Three comments this change falsified are corrected rather than left: two lazy
imports in models/ were justified by "import Auto3D.utils reaches this module",
which is no longer true, and would have invited someone to un-defer rdkit and
requests.

Verified: 1357 passed, 9 skipped, 1 xfailed, ruff clean; four seeds identical;
the documented order-sensitive trio green; all eight mutations confirmed red
and restored under sha256 check.
@isayev
isayev merged commit a1f1257 into main Aug 4, 2026
8 checks passed
@isayev
isayev deleted the debt/wave3-barrel-demolition branch August 4, 2026 13:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant