gh-152384: pixi-packages: use `flags` to define variants by lucascolley · Pull Request #152385 · python/cpython

lucascolley · 2026-06-27T10:56:40Z

cc @wolfv @baszalmstra @jaimergp I'm very excited about this PR!

Issue: pixi-packages: simplify to a single pixi.toml #152384
Issue: bump pixi-packages to 3.16 #152313

lucascolley · 2026-06-27T10:57:35Z

 - `freethreading`
 - `asan`: ASan-instrumented build
- `tsan-freethreading`: TSan-instrumented free-threading build
+- `tsan_freethreading`: TSan-instrumented free-threading build


I think it makes sense to remove the - from this variant name, given that it is invalid in the flags field: https://conda.org/learn/ceps/cep-0045#repodata-record-syntax

seems fine to me, I think you and @crusaderky are just about the only users of these builds so far for what its worth

lucascolley · 2026-06-27T10:59:04Z

    run_exports:
      noarch:
        - python
      weak:
        - python_abi ${{ version }}.* *_${{ abi_tag }}


we could also define flags on python_abi and perhaps not need to rely on abi_tag as much? But I suppose we still want to keep abi_tag around at least while conda-forge does?

EDIT: can probably leave this for a follow-up and open an issue

lucascolley · 2026-06-27T11:01:37Z

@StanFromIreland this goes some way towards making the version update situation better by reducing duplication in the variant definition

lucascolley · 2026-06-27T11:12:25Z

+openssl:
+  - '3.5'


ref. prefix-dev/pixi#6460 (comment) @h-vetinari, I wonder whether there is a better way to do this by pulling in (some of?) https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml

obviously you could download the current pinning file, yaml-load it and extract the openssl pin, but that's perhaps more complexity and dynamism than justified - an argument could be made that openssl is special enough to handle it separately... and although pins have a habit of going stale, staying on an LTS version probably makes sense here.

conda-forge will probably migrate to openssl 4 in the fall, but since AFAICT python has no other dependencies that are themselves openssl-dependent, this should not cause any conflicts (except in larger environments with the most recent builds, but that's not really relevant for the ASAN stuff).

obviously you could download the current pinning file, yaml-load it and extract the openssl pin, but that's perhaps more complexity and dynamism than justified

an idea was floated at some point that variant configs themselves could be packaged as conda packages and Pixi could consume them that way. Not sure if anyone has thought about that recently.

I mean, conda-forge-pinning is exactly that package. You'd still have to go parse the yaml though. 🤷

ngoldbaum · 2026-07-09T18:02:19Z

The CI failure looks unrelated but before this PR can be merged it's going to need a NEWS entry. Can you add one?

hugovk · 2026-07-10T11:39:45Z

I've updated this branch from main, which should pull in a skip for the failing unrelated macOS test.

lucascolley · 2026-07-10T13:40:04Z

news added

lucascolley · 2026-07-13T12:23:17Z

@ngoldbaum @kumaraditya303 any remaining concerns from your side? I think we can merge and patch up any more changes in a follow-up, this seems to be an improvement on the situation in main at least :)

Co-authored-by: Isuru Fernando <isuruf@gmail.com>

ngoldbaum

What happens if someone doesn't provide a variant flag? Which one is selected? None? Is there a good error? Should this be explicitly mentioned in the readme?

I left some more comments inline.

ngoldbaum · 2026-07-15T10:01:13Z

      - expat
      - if: linux and "san" in variant
        then:
          - libsanitizer


Is this still needed now that we're using clang everywhere?

I'm not sure, @isuruf do you know?

Okay yeah, I presume it is GCC-specific and thus not needed.

ngoldbaum · 2026-07-15T10:01:35Z

+openssl:
+  - '3.5'
+
+c_compiler:
+  - clang
+c_compiler_version:
+  - 21
+cxx_compiler:
+  - clangxx
+cxx_compiler_version:
+  - 21


can we add comments explaining these pin choices?

@kumaraditya303 how would you explain when these need to be updated, and where to look for the new versions?

@kumaraditya303 how would you explain when these need to be updated, and where to look for the new versions?

cc @vstinner @corona10

Who suggested it, and was it for to avoid a bug in newer clang, for reproducibility, or for building the JIT in the future (that needs LLVM 21, but the JIT isn't covered by the build variants today, and I'd think any recent version should work for default/free-threaded/asan/tsan builds)?

how would you explain when these need to be updated, and where to look for the new versions?

Ideally it should match the version of clang used in https://github.com/python/cpython/blob/main/.github/workflows/reusable-san.yml, so if that file is updated pixi definitions should be updated as well.

Co-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com>

lucascolley · 2026-07-15T12:17:19Z

What happens if someone doesn't provide a variant flag? Which one is selected? None? Is there a good error? Should this be explicitly mentioned in the readme?

Good catch. I think we may want to use down_prioritize_variant on non-default variants: https://rattler-build.prefix.dev/latest/variants/#prioritizing-variants

bedevere-app Bot mentioned this pull request Jun 27, 2026

pixi-packages: simplify to a single pixi.toml #152384

Open

bedevere-app Bot added the awaiting review label Jun 27, 2026

lucascolley commented Jun 27, 2026

View reviewed changes

lucascolley force-pushed the pixi-flags branch from 714d63f to 34cbfc3 Compare June 27, 2026 11:10

lucascolley commented Jun 27, 2026

View reviewed changes

pythongh-152384: pixi-packages: use flags to define variants

8b5b0c2

lucascolley force-pushed the pixi-flags branch from 34cbfc3 to 8b5b0c2 Compare June 27, 2026 11:16

lucascolley commented Jun 27, 2026

View reviewed changes

Comment thread Tools/pixi-packages/README.md Outdated

This was referenced Jun 27, 2026

ENH: pixi-packages: use skip-pyc-compilation for tsan-freethreading numpy/numpy#31558

Draft

ENH: pixi-packages: use flags numpy/numpy#31805

Draft

This was referenced Jul 4, 2026

BLD/DEV/CI: ASAN at your fingertips! scipy/scipy#24066

Open

GH-49321: [C++][Python] Add ASAN / UBSAN pixi builds for Arrow and PyArrow apache/arrow#49849

Open

ngoldbaum added awaiting changes and removed awaiting review labels Jul 9, 2026

Merge branch 'main' into pixi-flags

81c8972

lucascolley added 2 commits July 10, 2026 14:31

tweaks

b226f35

add news

5cba45c

lucascolley requested a review from ngoldbaum July 10, 2026 13:39

ngoldbaum added awaiting core review and removed awaiting changes labels Jul 10, 2026

lucascolley commented Jul 12, 2026

View reviewed changes

Comment thread Tools/pixi-packages/variants.yaml

pin to clang 21

f152ba7

isuruf reviewed Jul 13, 2026

View reviewed changes

Comment thread Tools/pixi-packages/variants.yaml Outdated

remove abi_tag from variant

18c12e6

Co-authored-by: Isuru Fernando <isuruf@gmail.com>

isuruf reviewed Jul 13, 2026

View reviewed changes

Comment thread Tools/pixi-packages/recipe.yaml Outdated

Update Tools/pixi-packages/recipe.yaml

44c27db

Co-authored-by: Isuru Fernando <isuruf@gmail.com>

isuruf reviewed Jul 14, 2026

View reviewed changes

Comment thread Tools/pixi-packages/README.md Outdated

Update README.md

a608b30

Co-authored-by: Isuru Fernando <isuruf@gmail.com>

ngoldbaum reviewed Jul 15, 2026

View reviewed changes

lucascolley and others added 3 commits July 15, 2026 13:30

revisions [skip ci]

8af45b3

Update Tools/pixi-packages/README.md

2b05522

Co-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com>

Update Tools/pixi-packages/build.sh

01a12c8

lucascolley mentioned this pull request Jul 15, 2026

gh-152313: bump pixi-packages to 3.16 #152314

Closed

down-prioritize non-default variants

08807bb

lucascolley requested a review from isuruf July 15, 2026 12:31

crusaderky mentioned this pull request Jul 15, 2026

Refactor llama-cpp recipes into 2 flag-based recipes crusaderky/pixi-llm-recipes#4

Merged

		openssl:
		- '3.5'

Uh oh!

Uh oh!

Conversation

lucascolley commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucascolley Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucascolley commented Jun 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ngoldbaum commented Jul 9, 2026

Uh oh!

hugovk commented Jul 10, 2026

Uh oh!

lucascolley commented Jul 10, 2026

Uh oh!

Uh oh!

lucascolley commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ngoldbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lucascolley commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

lucascolley commented Jun 27, 2026 •

edited

Loading

lucascolley Jun 27, 2026 •

edited

Loading