refactor!: delete the utils barrel and make utils a leaf package - #140
Merged
Conversation
Three findings that had to move together, since all three are about which module a name is imported from. utils/__init__.py re-exported 41 names. The recorded cause -- names missing from __all__ -- was stale: the file's imports and its __all__ were an exact bijection. The real defect is inconsistency. Five of eight submodules were re-exported and three were not, so energy, convergence and stereo_check had no barrel presence at all, while check_connectivity was reached through the barrel in filtering.py and through its own module in ranking.py, one file apart. The barrel is now docstring-only, documenting what each of the eight modules owns, and the 14 import lines across 12 files name the defining module. An AST test, scoped to src/Auto3D and not regex-based, asserts no first-party module imports through it again. batchopt.py's compat barrel was narrower than recorded: only print_stats was a pure re-export, and it is deleted. EnForce_ANI and n_steps are used in-module and stay, with a comment saying why they are not re-exports. Its consumers were exactly SPE.py and ASE/thermo.py; the three other sites in the finding import optimizing, which is defined there, and are untouched. utils/ is now a leaf. Both Auto3D.models imports in validation.py move to function scope, verified by a subprocess test asserting that importing Auto3D.utils.validation loads no Auto3D.models module. validation.py's module-level `import torch` is deliberately left where it is. 22 test sites patch Auto3D.utils.validation.torch.cuda.is_available, so deferring it breaks all 22 at once -- confirmed by actually deferring it and counting. A test pins the import scope so a future reader cannot discover this the expensive way. On the public surface: api.rst documents dotted paths, __all__ is the top-level barrel, and these are not the same set -- ten of api.rst's entries are Auto3D.exceptions.*, which do not belong at top level. So the rule enforced is one-directional: exported implies documented, and every documented path must resolve. Under it get_device, CustomNNP and IsomerEngineFactory stay documented at their own dotted paths rather than gaining top-level aliases, since a second supported path buys nothing and, for CustomNNP, would contradict the migration guide that names Auto3D.models.contract.CustomNNP as the survivor. generate_conformers is documented instead of removed: it has no defining module, existing only as the alias __init__.py calls canonical. Three comments this change falsified are corrected rather than left: two lazy imports in models/ were justified by "import Auto3D.utils reaches this module", which is no longer true, and would have invited someone to un-defer rdkit and requests. Verified: 1357 passed, 9 skipped, 1 xfailed, ruff clean; four seeds identical; the documented order-sensitive trio green; all eight mutations confirmed red and restored under sha256 check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wave 3: the
utilsbarrel is gone andutils/is a leaf package. Three findings that hadto move together, because all three are about which module a name is imported from.
utils/__init__.pyre-exported 41 namesThe recorded cause — "names missing from
__all__" — was stale. The file's imports andits
__all__were an exact bijection.The real defect is inconsistency. Five of eight submodules were re-exported and three were
not, so
energy,convergenceandstereo_checkhad no barrel presence at all. Meanwhilecheck_connectivitywas reached through the barrel infiltering.py:13and through its ownmodule in
ranking.py:12— two sibling files, two different paths to the same function.utils/__init__.pyis now docstring-only, documenting what each of the eight modules owns.The 14 barrel import lines across 12 files name the defining module instead. An AST test,
scoped to
src/Auto3D/**, asserts no first-party module imports through it again — ASTrather than regex, so it cannot be fooled by a mention in a comment or string.
Full 41-name mapping table (old barrel path → new dotted path) is in the CHANGELOG.
batchopt.py's compat barrel was narrower than recordedOnly
print_statswas a pure re-export; it is deleted.EnForce_ANIandn_stepsaregenuinely used in-module and stay imported, now with a comment saying they are not
re-exports so the next reader doesn't delete them.
Its consumers were exactly two:
SPE.py:9andASE/thermo.py:26, both now pointing atbatch_opt.model_wrapper. The finding also namedASE/geometry.py,auto3D.pyandworkflow_workers.py— those are wrong. They importoptimizing, which is definedthere, and are untouched.
utils/is now a leafBoth
Auto3D.models.*imports invalidation.pymove to function scope. It was the onlynon-leaf module in the package. A subprocess tripwire asserts that importing
Auto3D.utils.validationloads noAuto3D.modelsmodule — it was red for the stated reason,pulling in six of them.
import torchinvalidation.pystays at module level, deliberately22 test sites patch
Auto3D.utils.validation.torch.cuda.is_available. Deferring that importbreaks all 22 at once — confirmed by performing the mutation and counting, not by
reasoning about it. A test now pins the import scope so the next person doesn't rediscover
this the expensive way. M43 was about the
models/import, not about torch; conflating thetwo is the cheap-looking shortcut to an import-cost number that this test now blocks.
The public-surface rule, corrected
The rule proposed for this phase was "public iff in
Auto3D.__all__and inapi.rst".That is wrong, and implementing it literally would have been a mistake:
api.rstdocuments24 entries, ten of which are
Auto3D.exceptions.*, none of which belong at top level.api.rstdocuments dotted paths;__all__is the top-level barrel. They are not the sameset and never were.
So the implication is enforced one way only — exported ⟹ documented, plus every
documented path must resolve. Both directions are enforced by test, and
api.rstnow statesthe rule in prose.
__all__is unchanged at 13 names.Under that rule, the four names that were inconsistent resolve as:
get_deviceapi.rst-onlyAuto3D.model_factory.get_device(5× in the migration guide). Its apparent notebook uses aretorch.cuda.get_device_*, a different thing. A top-level alias would mint a second supported path for nothing.CustomNNPapi.rst-onlyfrom Auto3D.models import CustomNNPas the removed form andAuto3D.models.contract.CustomNNPas the survivor. A third path would contradict it.IsomerEngineFactoryapi.rst-onlyisomers/__init__.py's docstring already teaches.generate_conformersapi.rst__init__.pycalls canonical. Deleting the intended-canonical name to satisfy a docs list is the wrong direction.Found while implementing, and not swept under the rug
Two more package barrels violate the rule:
cli/__init__.py(6 names) andmodels/__init__.py(7 names). Neither is documented at its package path. Both are out ofscope here, so
test_only_documented_subpackages_define_allfreezes the allowed set at{isomers, cli, models}with a comment recording the latter two as debt rather thanblessing them. Removing one means deleting it from that frozen set, and nobody can add a
fourth by accident.
Three comments this change falsified, corrected minimally — mechanism updated, conclusion
kept. Two lazy imports (
models/species.py,models/preflight.py) were justified by"
import Auto3D.utilsreaches this module", which is no longer true, as was a test docstring.Left as-is they would have invited someone to un-defer
rdkitandrequestsand quietlyundo wave 1's import-cost win.
Verification
E_totfilter divergence, still owned by a later cluster.(
test_lazy_torchani_import→test_torch_config→test_thermo_helpers): 111 passed.This is the change most likely to break module identity, since it rewrites import sources
across 12 modules.
verified red and restored under sha256 check.