SG-43217 Integrate Flow Data SDK as a vendored beta library#1098
Open
stevelittlefish wants to merge 11 commits into
Open
SG-43217 Integrate Flow Data SDK as a vendored beta library#1098stevelittlefish wants to merge 11 commits into
stevelittlefish wants to merge 11 commits into
Conversation
Adds requirements/any/ for Python-version-independent vendor zips and teaches python/tank_vendor/__init__.py to auto-discover and load them alongside the existing pkgs.zip. Drops in flow_data_sdk-beta.zip as the first such vendor. The loader refactor extracts the existing pkgs.zip init into a reusable _load_packages_from_zip helper. Shared zips load after pkgs.zip so per-version pins win on name collision; collisions warn and skip rather than overwrite. Per-package import failures continue to warn-and-continue (the SDK uses 3.10+ syntax, so it'll simply be absent on 3.7/3.9 instead of breaking import tank_vendor). Includes a small _patch_flow_data_sdk_version workaround for an upstream bug: the SDK's _version.py queries importlib.metadata.version( "adsk-flow-data") but the published wheel's distribution name is "flow-data-sdk", so SDK_VERSION otherwise falls back to "local_dev" even with .dist-info present. The patch is a self-disabling no-op once upstream is fixed. Tests cover the new package via PACKAGES_TO_TEST (3.10+ gated) plus a TestFlowDataSDK class with a dist-info canary that catches future regressions in the zip's metadata packaging. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Flow Data SDK previously hardcoded the wrong distribution name in
flow_data_sdk/base/_version.py ("adsk-flow-data" instead of the actual
published name "flow-data-sdk"), so importlib.metadata.version() always
raised PackageNotFoundError and SDK_VERSION fell back to "local_dev"
even with .dist-info present in our shared zip.
The new SDK zip ships the upstream one-line fix — _version.py now
queries the correct name and SDK_VERSION resolves to the real version
on its own. The local workaround patch can go.
Removes:
- _patch_flow_data_sdk_version function and its call site in
tank_vendor/__init__.py (~37 lines)
- The upstream-bug commentary in the test_dist_info_via_importlib_metadata
docstring (the assertion itself is unchanged and still passes)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1098 +/- ##
==========================================
+ Coverage 79.42% 79.51% +0.09%
==========================================
Files 198 198
Lines 20750 20842 +92
==========================================
+ Hits 16481 16573 +92
Misses 4269 4269
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
The required parameter was only meaningful in the late-stage-exception branch (raise vs warn). The other two failure paths returned False identically in both modes, so the parameter was mostly dead weight. Collapse to always-raise for any zip's wholesale load failure, matching the original pkgs.zip posture. Shared zips in requirements/any/ are now held to the same standard: a corrupt or import-broken shared zip will fail import tank_vendor with a clean RuntimeError instead of silently degrading. Per-package ImportError inside the loop still warns and skips, so flow_data_sdk being absent on Python 3.7/3.9 (3.10+ syntax) remains non-fatal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes a regression in the share_core integration test on Windows / Python 3.13. flow_data_sdk's _version.py calls importlib.metadata.version( "flow-data-sdk") at import time. importlib.metadata iterates sys.path in order, and FastPath.zip_children() is @lru_cache'd — the cached FastPath holds an open zipfile.ZipFile to whichever zip it probed. Previously pkgs.zip was at sys.path[0] and flow_data_sdk-beta.zip at sys.path[1], so the scan probed pkgs.zip first (no match → cached open handle), then matched in flow_data_sdk-beta.zip. The lingering handle on pkgs.zip caused share_core's shutil.move to fail with WinError 32 when relocating install/core on Windows. Reorder so shared zips end up at sys.path[0], pkgs.zip at sys.path[1]. importlib.metadata then short-circuits on the first probe and never opens pkgs.zip. pkgs.zip is still loaded into sys.modules first, so collision precedence is unchanged. Drop the path_position parameter from _load_packages_from_zip — every zip is now always inserted at sys.path[0], and the call order in tank_vendor/__init__.py determines the final sys.path ordering. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… zips The previous reorder commit (f96c7d6) moved the share_core WinError 32 from pkgs.zip to flow_data_sdk-beta.zip — same root cause, different file. This is the actual fix. importlib.metadata.FastPath.__new__ is @lru_cache'd. The FastPath instance for whichever zip importlib.metadata probes is kept alive forever, and inside FastPath.zip_children() the line `self.joinpath = zip_path.joinpath` binds a zipfile.Path (with its underlying open ZipFile) as an instance attribute. The result: the cache permanently pins an open file handle on every zip that ever yielded a metadata match. flow_data_sdk's _version.py triggers this by calling importlib.metadata.version("flow-data-sdk") at module import time. The cached FastPath then keeps flow_data_sdk-beta.zip open, which on Windows blocks share_core's shutil.move(install/core, ...). Fix: after all zips are loaded, call MetadataPathFinder().invalidate_caches() to drop FastPath references, then gc.collect() so the underlying ZipFile.__del__ fires immediately and releases the OS handle. invalidate_caches is called on an instance, not the class, because it isn't decorated as @classmethod in older Python versions but takes `cls` by convention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cleanup added in 8c006ca (clearing FastPath cache + gc.collect to release zipfile handles) is a workaround for Windows' sharing-violation file-move semantics. Linux and macOS allow moving files with open handles, so the cleanup is unnecessary there — and it was observed to break a Linux / Python 3.13 integration test in CI. Guard with sys.platform == "win32" so non-Windows platforms get the previous behaviour unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three changes prompted by PR review on #1098: 1. Broaden per-package import catch from ImportError to Exception. The inner try/except is best-effort by design (the outer wholesale-failure handler still raises). A future shared vendor using PEP 604 unions or other syntax-level newness would currently break import tank_vendor on older Pythons with a SyntaxError escaping the inner catch; widening the except matches the documented intent. 2. Add TestFlowDataSDKAbsentOnOldPython, gated to Python < 3.10. Pins the contract that the loader warns and continues when a shared vendor fails to import, so the PR's behavioural claim ("on 3.7/3.9 the SDK is simply absent") is actually exercised in CI rather than just asserted in the description. 3. Soften the misleading "mandatory" label on pkgs.zip in the module docstring. Missing pkgs.zip is tolerated to support pip-installed tk-core where dependencies come from the environment; the docstring for _load_packages_from_zip already says so, but the top-of-file summary still claimed otherwise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
carlos-villavicencio-adsk
left a comment
carlos-villavicencio-adsk
left a comment
There was a problem hiding this comment.
Looks good so far.
I'd love to see python-api (shotgun_api3) also in the requirements/any because it's also a python version independent. I'm not suggesting this for the PR, it can be done later.
- Filter top-level .dll files in _discover_top_level_packages so a stray Windows DLL at the root of a vendor zip is not treated as an importable package (Carlos). - Reword the unreadable-zip warning to acknowledge that affected dependencies may still resolve from the Python environment instead of implying a guaranteed failure (Copilot). - Pass RuntimeWarning + stacklevel=2 on per-package import failures so they match the other warnings in this module and point at the caller (Copilot). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
julien-lang
reviewed
May 19, 2026
- Restore the unicode → arrow in the _install_import_hook docstring. - Wrap the new requirements/any/ paragraph in developer/README.md. - Restore the Step 1/2/3/4 navigational comments inside _load_packages_from_zip that the refactor had stripped. - Drop the Python<3.8 fallback in _release_importlib_metadata_handles; Python 3.7/3.8 compatibility was discontinued after March 2026. - Reorganise __init__.py so all helper defs (including _release_importlib_metadata_handles) come before the MAIN INITIALIZATION block, and call the release helper from the main block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
requirements/any/for Python-version-independent vendor zips, sibling to the existing per-Python-versionpkgs.zipdirectories.python/tank_vendor/__init__.pyto auto-discover and load every*.zipinrequirements/any/alongsidepkgs.zip, aliasing each top-level package undertank_vendor.*via the existing meta-path finder.flow_data_sdk-beta.zipas the first such shared vendor — the Autodesk Flow Data SDK at2.0.37, used by Toolkit code asfrom tank_vendor import flow_data_sdk.Implementation notes
pkgs.zipinit was extracted into a reusable_load_packages_from_ziphelper.pkgs.zipraisesRuntimeErroronly on a wholesale import failure inside the zip; missing or unreadablepkgs.zipis tolerated (warns for unreadable; silent for missing, to support pip-installed setups where dependencies come from the Python environment). Shared zips warn-and-continue throughout.pkgs.zipis loaded first, so per-Python-version pins win on name collision. Collisions warn and skip rather than overwrite.types.UnionType,typing.TypeAlias); on 3.7/3.9 it will simply be absent fromtank_vendorinstead of breakingimport tank_vendor._patch_shotgun_api3_certsis untouched.Files
python/tank_vendor/__init__.py_load_packages_from_ziphelper; addrequirements/any/glob looprequirements/any/flow_data_sdk-beta.ziptests/core_tests/test_tank_vendor.pyPACKAGES_TO_TEST(3.10+ gated) + newTestFlowDataSDKclassdeveloper/README.mdrequirements/any/locationTest plan
python tests/run_tests.py core_tests/test_tank_vendor.pypasses (15 tests; locally verified on 3.10)python tests/run_tests.py core_testspasses with no regressions (481 tests; locally verified on 3.10)import tank_vendorsucceeds and emits a single warning thatflow_data_sdkisn't available (expected; SDK requires 3.10+)python -c "from tank_vendor import flow_data_sdk; print(flow_data_sdk.SDK_VERSION); from tank_vendor.flow_data_sdk import GQLClient; print(GQLClient)"prints2.0.37and theGQLClientclass🤖 Generated with Claude Code