Skip to content

[runtime][python] Refresh imports for new dependency generations - #943

Draft
joeyutong wants to merge 1 commit into
apache:mainfrom
joeyutong:codex/fix-python-dependency-generation
Draft

[runtime][python] Refresh imports for new dependency generations#943
joeyutong wants to merge 1 commit into
apache:mainfrom
joeyutong:codex/fix-python-dependency-generation

Conversation

@joeyutong

Copy link
Copy Markdown
Collaborator

Linked issue: #941

Purpose of change

Flink may rematerialize a job's Python dependencies under a new python-dist-* directory after task failover, while Pemja's process-level main interpreter and its import state remain alive in the TaskManager JVM. Existing user modules can therefore keep __file__, __path__, or __spec__ entries that point to the removed dependency directory.

This change activates each job's current dependency generation before loading Python actions or resources. When the generation changes, it:

  • clears the Flink Agents Python function cache;
  • evicts modules attributed to the previous generation from sys.modules;
  • removes stale sys.path and importer-cache entries;
  • prepends the current generation's PYTHONPATH entries; and
  • preserves modules and paths owned by other active jobs.

The generation is recorded only after the refresh succeeds, so a failed refresh is retried on the next initialization.

Tests

  • uv run --project python pytest -q python/flink_agents/runtime/tests/test_python_dependency.py (3 passed)
  • mvn -B --no-transfer-progress -pl runtime -am -Dtest=PythonDependencyGenerationManagerTest -Dsurefire.failIfNoSpecifiedTests=false test (1 passed)
  • Maven Spotless formatting/checks passed as part of the Java verification

API

No public API changes.

Documentation

  • doc-needed
  • doc-not-needed
  • doc-included

AI-Contributed/Feature: 0/242
AI-Contributed/UT: 0/257
@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue. labels Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant