Skip to content

chore(ci): enable Python 3.15 support for dramatiq - #19909

Open
wantsui wants to merge 20 commits into
mainfrom
codex-dramatiq-py315
Open

chore(ci): enable Python 3.15 support for dramatiq#19909
wantsui wants to merge 20 commits into
mainfrom
codex-dramatiq-py315

Conversation

@wantsui

@wantsui wantsui commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Description

Builds on #19879, which already enabled Dramatiq testing on Python 3.14 and added the 17f991e lockfile. This PR will unblock the integrations CI for dramatiq (and later other integrations).

This PR enables the next runtime only:

  • Adds Python 3.15 only to the Dramatiq Riot environment; the global repository runtime list remains capped at Python 3.14 until closer to the Python 3.15 release.
  • Extends the latest-package Dramatiq environment from Python 3.14 through Python 3.15. This is our test to make sure CI isn't totally broken.
  • Adds the generated Dramatiq Python 3.15 lockfile via a container using the piptools that work for Python 3.15
    Related PRs:
    ci(testrunner): install pip-tools for Python 3.15 #19977
    ci: update testrunner image #19984

Testing

  • scripts/lint fmt -- riotfile.py
  • scripts/lint riot
  • scripts/lint checks
  • scripts/run-tests --list tests/contrib/dramatiq/ confirms the existing Python 3.14 venv 17f991e and new Python 3.15 venv 4112544
  • Regenerated 4112544 with Riot under Python 3.15; the lockfile was unchanged

Risks

Low integration risk because this changes only Dramatiq Riot coverage and generated metadata. Python 3.15 remains enabled per integration while the global matrix stays capped at 3.14.

Additional Notes

  • Python 3.14 support was merged separately in test(dramatiq): enable tests for Python 3.14 #19879 and is not added by this PR.
  • Base branch: main.
  • Local Python 3.15 installation currently requires the documented PIP_IGNORE_REQUIRES_PYTHON=1 override because the package metadata still declares Requires-Python <3.15.

Historical CI error for 3.15

Issue 1: Injection Assembly errors (fixed in this PR)
There's 174 failures at the moment:
image

Based on https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/1995599108 , we can't use INJECTION_ASSEMBLY

^^^^^^^^^
  File "/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/coverage/instrumentation_py3_12.py", line 19, in <module>
    from ddtrace.internal.coverage.import_instrumentation_py3_12 import ImportName
  File "/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/coverage/import_instrumentation_py3_12.py", line 9, in <module>
    from ddtrace.internal.bytecode_injection import INJECTION_ASSEMBLY
ImportError: cannot import name 'INJECTION_ASSEMBLY' from 'ddtrace.internal.bytecode_injection' (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/internal/bytecode_injection/__init__.py)
Test failed with exit code 1

Issue 2: Lock files
Regenerated after the pip-tools PR.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codeowners resolved as

Resolved from the full PR diff against vlad/py315-riotfile using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

.riot/requirements/128b106.txt                                          @DataDog/apm-python
.riot/requirements/1381214.txt                                          @DataDog/apm-python
.riot/requirements/14116fa.txt                                          @DataDog/apm-python
.riot/requirements/17f991e.txt                                          @DataDog/apm-python
.riot/requirements/19508cd.txt                                          @DataDog/apm-python
.riot/requirements/4112544.txt                                          @DataDog/apm-python
.riot/requirements/638973a.txt                                          @DataDog/apm-python
ddtrace/internal/coverage/import_instrumentation_py3_12.py              @DataDog/ci-app-libraries
ddtrace/internal/coverage/instrumentation_py3_12.py                     @DataDog/ci-app-libraries
riotfile.py                                                             @DataDog/apm-python
scripts/integration_registry/registry.yaml                              @DataDog/apm-idm-python
supported_versions.json                                                 @DataDog/apm-core-python @DataDog/apm-idm-python
tests/coverage/test_import_dependency_tracking.py                       @DataDog/ci-app-libraries
tests/coverage/test_import_instrumentation_py312.py                     @DataDog/ci-app-libraries

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Aug 27, 2026

Copy link
Copy Markdown

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 230 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 230 highest severity)
ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.llmobs._integrations.openai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.profiling.collector.pytorch -×-> ddtrace.trace  (product:profiling -> product:tracing, score=132)
ddtrace.appsec._listeners -×-> ddtrace.trace  (product:appsec -> product:tracing, score=132)
ddtrace.internal.openfeature._span_enrichment -×-> ddtrace.trace  (product:openfeature -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Aug 27, 2026

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 3 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

@datadog-official

This comment has been minimized.

@pr-commenter

pr-commenter Bot commented Aug 27, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-09-04 21:34:38

Comparing candidate commit 2542bca in PR branch codex-dramatiq-py315 with baseline commit 8c7799f in branch vlad/py315-riotfile.

📊 Benchmarking dashboard

Found 0 performance improvements and 8 performance regressions! Performance is the same for 577 metrics, 10 unstable metrics, 3 known flaky benchmarks, 15 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationinject-ids_only

  • 🟥 execution_time [+2.026µs; +2.207µs] or [+11.887%; +12.947%]

scenario:iastaspects-join_aspect

  • 🟥 execution_time [+44.982µs; +48.478µs] or [+20.568%; +22.167%]

scenario:iastaspects-title_aspect

  • 🟥 execution_time [+64.402µs; +69.707µs] or [+23.485%; +25.420%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+118.694µs; +126.103µs] or [+28.713%; +30.505%]

scenario:iastaspectssplit-rsplit_aspect

  • 🟥 execution_time [+15.697µs; +20.958µs] or [+11.199%; +14.952%]

scenario:samplingrules-high_match

  • 🟥 execution_time [+11.957µs; +13.678µs] or [+7.624%; +8.720%]

scenario:telemetryaddmetric-1-count-metric-1-times

  • 🟥 execution_time [+496.043ns; +535.804ns] or [+17.899%; +19.333%]

scenario:tracer-small

  • 🟥 execution_time [+37.287µs; +40.410µs] or [+11.435%; +12.393%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

  • unstable execution_time [-686.761ns; +778.028ns] or [-6.206%; +7.031%]

scenario:coreapiscenario-core_dispatch_1_listener

  • unstable execution_time [-29.965ns; +35.767ns] or [-4.918%; +5.870%]

scenario:coreapiscenario-core_dispatch_50_listeners

  • unstable execution_time [-1639.219ns; +1650.695ns] or [-9.647%; +9.714%]

scenario:coreapiscenario-core_dispatch_exception_listeners

  • unstable execution_time [-1298.467ns; +1189.946ns] or [-9.938%; +9.107%]

scenario:coreapiscenario-core_dispatch_listeners

  • unstable execution_time [-328.056ns; +322.495ns] or [-8.954%; +8.802%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

  • unstable execution_time [-251.773ns; +260.930ns] or [-8.636%; +8.950%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

  • unstable execution_time [-81.865ns; +66.627ns] or [-7.118%; +5.793%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

  • unstable execution_time [-4021.878ns; +4032.007ns] or [-9.764%; +9.789%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

  • unstable execution_time [-761.311ns; +789.127ns] or [-9.403%; +9.747%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

  • unstable execution_time [-58.278µs; +53.172µs] or [-9.979%; +9.105%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:iastaspects-casefold_noaspect

  • 🟥 execution_time [+39.950µs; +44.668µs] or [+16.334%; +18.263%]

scenario:iastaspects-ljust_noaspect

  • 🟥 execution_time [+41.965µs; +48.514µs] or [+14.238%; +16.460%]

scenario:span-start

  • 🟥 execution_time [+1.581ms; +1.748ms] or [+10.961%; +12.120%]

Known flaky benchmarks without significant changes:

  • scenario:errortrackingflasksqli-baseline
  • scenario:flasksimple-iast-get
  • scenario:iastaspects-casefold_aspect
  • scenario:iastaspects-index_aspect
  • scenario:iastaspects-lower_aspect
  • scenario:iastaspects-replace_aspect
  • scenario:iastaspects-swapcase_aspect
  • scenario:iastaspects-title_noaspect
  • scenario:iastaspects-translate_aspect
  • scenario:iastaspects-translate_noaspect
  • scenario:iastaspects-upper_noaspect
  • scenario:packagespackageforrootmodulemapping-cache_off
  • scenario:packagespackageforrootmodulemapping-cache_on
  • scenario:sethttpmeta-all-enabled
  • scenario:telemetryaddmetric-record-100-metrics

@wantsui
wantsui changed the base branch from vlad/315-official-support to vlad/py315-riot-install August 28, 2026 17:59
@wantsui
wantsui changed the base branch from vlad/py315-riot-install to vlad/315-official-support August 28, 2026 18:07
@wantsui
wantsui changed the base branch from vlad/315-official-support to vlad/py315-riot-install August 28, 2026 18:10
@wantsui
wantsui force-pushed the codex-dramatiq-py315 branch from 9f96d0a to 57afb96 Compare August 28, 2026 18:11
@vlad-scherbich
vlad-scherbich force-pushed the vlad/py315-riot-install branch from 801e339 to 19654b8 Compare August 28, 2026 18:34
Comment thread tests/coverage/test_import_instrumentation_py312.py
Comment thread ddtrace/internal/coverage/import_instrumentation_py3_12.py
@vlad-scherbich
vlad-scherbich force-pushed the vlad/py315-riot-install branch from 530a2f5 to 680d8a6 Compare August 31, 2026 18:31
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Sep 1, 2026
## Description

This PR was done with codex.

While trying to enable CI for Python 3.15 support in #19909, I came across a bunch of tests that failed because they are missing the lock file/requirements.


Note that Codex found that we have to use `pip<25.3` for now because newer versions of pip with `pip-tools 7.5.2` (what dd-trace-py uses right now), it throws `ImportError: cannot import name stdlib_pkgs from pip._internal.utils.compat`.


## Testing

- Built `docker/Dockerfile` locally successfully.
- Verified `python3.15 -m piptools compile --help` succeeds in the rebuilt image.

## Risks

None to tracer runtime code. This changes only testrunner build tooling.

## Additional Notes

This is temporary until Python 3.15 is out.

Co-authored-by: vlad.scherbich <vlad.scherbich@datadoghq.com>
@vlad-scherbich
vlad-scherbich force-pushed the vlad/py315-riot-install branch 3 times, most recently from c224d64 to 24f53a0 Compare September 2, 2026 02:20
Base automatically changed from vlad/py315-riot-install to main September 2, 2026 03:41
@wantsui
wantsui changed the base branch from main to vlad/315-peripheral-compat September 2, 2026 20:14
@vlad-scherbich
vlad-scherbich force-pushed the vlad/315-peripheral-compat branch from 4d0ca17 to 746d444 Compare September 2, 2026 20:14
@wantsui wantsui changed the title chore(dramatiq): enable Python 3.15 support chore(integrations): enable Python 3.15 support for integrations CI Sep 3, 2026
@wantsui
wantsui force-pushed the codex-dramatiq-py315 branch from 9029a3c to 964e4f5 Compare September 3, 2026 20:11
@wantsui wantsui added AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos changelog/no-changelog A changelog entry is not required for this PR. labels Sep 3, 2026
@wantsui wantsui changed the title chore(integrations): enable Python 3.15 support for integrations CI chore(ci): enable Python 3.15 support for CI Sep 3, 2026
@wantsui wantsui changed the title chore(ci): enable Python 3.15 support for CI chore(ci): enable Python 3.15 support for integrations CI Sep 3, 2026
@wantsui
wantsui force-pushed the codex-dramatiq-py315 branch from 964e4f5 to afdea50 Compare September 3, 2026 20:40
@wantsui wantsui changed the title chore(ci): enable Python 3.15 support for integrations CI chore(ci): enable Python 3.15 support for dramatiq Sep 3, 2026
@wantsui
wantsui marked this pull request as ready for review September 3, 2026 21:34
vlad-scherbich and others added 3 commits September 4, 2026 13:40
Extends the latest-package dramatiq Riot environment from Python 3.14
through 3.15 (3.9-3.14 coverage preserved), adds the generated 3.15
lockfile, and fixes the Python 3.12+ coverage/bytecode-injection
instrumentation module so it imports cleanly under 3.15.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@wantsui
wantsui force-pushed the codex-dramatiq-py315 branch from 79863b6 to e5db292 Compare September 4, 2026 18:10
These lines are leftovers from an unrelated PR stack and don't belong
in the dramatiq Python 3.15 support change.
Comment thread ddtrace/internal/coverage/instrumentation_py3_12.py Outdated
These weren't meant as durable AIDEV anchors, just explanatory comments.
Comment thread ddtrace/internal/coverage/import_instrumentation_py3_12.py Outdated
Comment thread ddtrace/internal/coverage/import_instrumentation_py3_12.py Outdated
Comment thread ddtrace/internal/coverage/import_instrumentation_py3_12.py Outdated
Comment thread ddtrace/internal/coverage/import_instrumentation_py3_12.py Outdated
Comment thread ddtrace/internal/coverage/import_instrumentation_py3_12.py
Comment thread ddtrace/internal/coverage/instrumentation_py3_12.py Outdated
Comment thread tests/coverage/test_import_dependency_tracking.py
Comment thread riotfile.py Outdated
wantsui and others added 2 commits September 4, 2026 15:18
Co-authored-by: Vlad Scherbich <vlad.scherbich@datadoghq.com>
Co-authored-by: Vlad Scherbich <vlad.scherbich@datadoghq.com>

@emmettbutler emmettbutler left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good aside from the huge comment

Comment thread ddtrace/internal/coverage/instrumentation_py3_12.py Outdated
- inline _decoded_import_name into iter_import_events (single consumer)
- slim verbose comment explaining the 3.15 injection gap
- mark test skip reason with TODO(py-315)
…-dramatiq-py315-clean-tmp

# Conflicts:
#	ddtrace/internal/coverage/import_instrumentation_py3_12.py
@wantsui
wantsui changed the base branch from main to vlad/py315-riotfile September 4, 2026 20:17
Base is now vlad/py315-riotfile, which adds (3, 15) to
SUPPORTED_PYTHON_VERSIONS, so select_pys(max_version="3.15") already
covers the full 3.9-3.15 range.

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The new Python 3.15 Dramatiq lock pins 2.2.0. The Riot environment requests the latest release, and the other changed lockfiles and registry use 2.2.1.

Open Bits AI session

🤖 Datadog Autotest · Commit 06c5be1 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Comment thread .riot/requirements/4112544.txt Outdated
The lock generated when 3.15 support was added still pinned dramatiq
2.2.0, one release behind the 2.2.1 lock used on 3.9-3.14. Flagged by
Datadog Autotest.
Base automatically changed from vlad/py315-riotfile to main September 4, 2026 21:22

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The Python 3.15 Dramatiq matrix agrees with the generated dependency files and the coverage fallback. The static review identifies no concrete regression.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit 2542bca · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos changelog/no-changelog A changelog entry is not required for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants