Skip to content

Add team_name to DagRunProtocol to match the DagRun Pydantic model. - #68403

Open
dkranchii wants to merge 2 commits into
apache:mainfrom
dkranchii:sync-dagrunprotocol-team-name
Open

Add team_name to DagRunProtocol to match the DagRun Pydantic model.#68403
dkranchii wants to merge 2 commits into
apache:mainfrom
dkranchii:sync-dagrunprotocol-team-name

Conversation

@dkranchii

@dkranchii dkranchii commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

DagRun.team_name was added in #65617 ("Add team name to task CTX") and is
exposed to user code via context["dag_run"].team_name and the
AIRFLOW_CONTEXT_TEAM_NAME env var, but DagRunProtocol (the type used
for Context["dag_run"]) was last synced before that change in #65347 and
did not pick the field up. Type checkers therefore flag
context["dag_run"].team_name as [attr-defined] even though the access
is correct at runtime.

Follow-up to #65347 / #67216, same pattern.


Important

🛠️ Maintainer triage note for @dkranchii · by @potiuk · 2026-07-08 15:38 UTC

Helpful heads-up from the maintainers — please address before this PR can be reviewed:

  • Pre-commit / static checks failing (CI image checks / Static checks). See the contributor guide.

Full criteria: Pull Request quality criteria.

The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.

Automated triage — may be imperfect; a maintainer takes the next look.

@dkranchii
dkranchii force-pushed the sync-dagrunprotocol-team-name branch from 6ea808d to dc47cdb Compare June 11, 2026 19:25
@vincbeck
vincbeck requested a review from o-nikolas June 11, 2026 19:38
@eladkal
eladkal force-pushed the sync-dagrunprotocol-team-name branch from dc47cdb to fcda014 Compare June 12, 2026 07:12
@eladkal eladkal added this to the Airflow 3.3.0 milestone Jun 12, 2026
@dkranchii

Copy link
Copy Markdown
Contributor Author

Test Result:
pr_test

amoghrajesh
amoghrajesh previously approved these changes Jun 15, 2026

@amoghrajesh amoghrajesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI needs fixing.

@eladkal

eladkal commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

@dkranchii you will need to run pre-commits to generate the missing files and then commit them

@dkranchii
dkranchii force-pushed the sync-dagrunprotocol-team-name branch from fcda014 to 5fdd83e Compare June 18, 2026 05:18
@dkranchii

Copy link
Copy Markdown
Contributor Author

@eladkal please review.

consumed_asset_events: list[AssetEventDagRunReference]
partition_key: str | None
note: str | None
team_name: str | None

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like the wrong place to expose this. team_name is not a property of the dag run, but of the DAG.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashb You're right that team_name is semantically a property of the Dag (resolved through its bundle), not of the run - every DagRun of the same Dag has the same team_name, and there's no team_name column on the DagRun ORM either.

Where it does already live is the server-side Execution API DagRun payload - it was added there (see AddTeamNameField in execution_api/versions/v2026_06_16.py) so it can be delivered as part of TIRunContext at task start, saving the worker an extra get_dag() round trip. This PR is just keeping DagRunProtocol in sync with that server schema so SDK consumers reading dag_run.team_name don't hit a type-checker failure.

I've pushed a comment on the field that spells this out - the denormalization is a deliberate choice at the server-schema layer, not something we're claiming is the "true" home of the field. If you'd rather see this promoted to DagResponse (and thus DagResult on the SDK side) with team_name removed from DagRun, happy to open a follow-up that does that server-side refactor - but I think it's worth keeping the SDK protocol in sync with the server as long as the server exposes it here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, basically this was the cleanest way to plumb the team_name to the context on the worker side.

This feels like the wrong place to expose this. team_name is not a property of the dag run, but of the DAG.

This isn't even true, I wish it was, but the community would not allow it to be a property of DAG or Task (or indeed DagRun) when designing the multi team feature. It is only a property of the Dag Bundle, everything else (Dag, Dag Run, task, etc) all just infer from Bundle.

dag_run sub model on the TIRunContext already has conf, notes, triggering_user_name, etc and it was the closest thing that maps to team_name. And adding a whole new model just for team_name is way over kill. And there is already code that iterates over dag_run and exports the AIRFLOW_CTX_* env vars.

tl;dr: It was the closest match without adding hundreds of lines of code and new mechanisms just for team, this PR is just updating typing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashb Given the design context @o-nikolas laid out above - team lives on the bundle and DagRun sub-model on TIRunContext is the pragmatic carrier that already ships from the server - this PR is purely a typing sync with the already-shipped Execution API schema. If moving team_name off DagRun server-side is desirable, happy to file a follow-up issue. If you're OK with landing this typing sync as-is, could you resolve the thread?

@vatsrahul1001

Copy link
Copy Markdown
Contributor

@dkranchii can you address open conversations

@dkranchii
dkranchii force-pushed the sync-dagrunprotocol-team-name branch 2 times, most recently from 4091fd1 to 0515f95 Compare July 1, 2026 19:44
@eladkal eladkal added the backport-to-v3-3-test Backport to v3-3-test label Jul 1, 2026
Comment thread task-sdk/src/airflow/sdk/types.py Outdated
@dkranchii
dkranchii force-pushed the sync-dagrunprotocol-team-name branch 2 times, most recently from 1fa7658 to 8078a70 Compare July 7, 2026 22:19
@o-nikolas

Copy link
Copy Markdown
Contributor

@dkranchii it looks like just static check failures remain, can you address those?

@dkranchii
dkranchii force-pushed the sync-dagrunprotocol-team-name branch from 8078a70 to 40444aa Compare July 9, 2026 19:10
dkranchii and others added 2 commits July 9, 2026 14:49
@dkranchii
dkranchii force-pushed the sync-dagrunprotocol-team-name branch from 40444aa to 8fde52a Compare July 9, 2026 22:11
@dkranchii

Copy link
Copy Markdown
Contributor Author

@o-nikolas can you review this pr.

@dkranchii

Copy link
Copy Markdown
Contributor Author

@ashb @vincbeck can you pls review this pr.

@vincbeck

Copy link
Copy Markdown
Contributor

@o-nikolas @ashb ?

@dkranchii

Copy link
Copy Markdown
Contributor Author

@ashb all checks have passed and the PR has received approval. If everything looks good, i'd appreciate it if this PR could be merged. Thank you !

@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 18, 2026
@vatsrahul1001
vatsrahul1001 requested a review from ashb August 3, 2026 05:45
@amoghrajesh
amoghrajesh dismissed their stale review August 4, 2026 10:02

Ash's concern seems valid there, so I am revoking my approval

@ashb ashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm missing something - i don't even understand why we need to pass this from the server at all.

In multi team, a worker belongs to exactly one team doesn't it? So the worker process/config should also know its team name.

What am i missing/misunderstanding

@dkranchii

Copy link
Copy Markdown
Contributor Author

@ashb

Fair question - the intuition breaks down in three places, so it took me a bit to unpack too.

1. team_name is a property of the Dag Bundle, not the worker.

The design that landed for multi-team makes the relationship
Task → Dag → Dag Bundle → Team (see
airflow-core/docs/core-concepts/multi-team.rst).
Team lives on the bundle association only; DAG, DagRun, TaskInstance, worker,
etc. all infer it from the bundle. That was a deliberate community
decision when the feature was designed - team_name is not a first-class
attribute of any of the runtime entities.

2. Not every worker/executor is per-team, and even where it is, the task
subprocess doesn't inherit that arg.

  • airflow tasks run (the process that actually executes user code) has
    no --team-name argument - task subprocesses don't get one, regardless
    of executor.
  • KubernetesExecutor launches a pod per task via airflow tasks run;
    the pod has no team CLI flag.
  • LocalExecutor runs in the scheduler process; the scheduler is
    process-wide, not team-scoped. Team-scoped LocalExecutors are just
    another instance inside the same scheduler.
  • CeleryExecutor does have airflow celery worker --team on the worker
    parent
    , but the task subprocess it spawns still doesn't see it.
  • EdgeWorker has --team-name, but the docstring on that arg spells it
    out - "This is a UI/REST API-level hint; the Execution API does not
    currently enforce team-based access boundaries… If omitted, the worker
    operates as a default-team worker."
    (providers/edge3/src/airflow/providers/edge3/cli/definition.py)

So "the worker process/config knows its team" is only true for Celery+Edge
worker parents, and even there the value doesn't reach the task
subprocess through any existing mechanism.

3. The server is the authoritative source, and it already ships this.

get_team_name_for_ti in
airflow-core/src/airflow/api_fastapi/execution_api/security.py
resolves the task's team from TaskInstance → DagModel → DagBundleModel → Team and the ti_run route stamps it onto DagRun.team_name in
TIRunContext. That happened in
#65617 (Add team name to task CTX)
with a versioned schema change (AddTeamNameField in
execution_api/versions/v2026_04_17.py), and the SDK's generated
_generated.py DagRun model already carries the field. It's what
populates AIRFLOW_CTX_TEAM_NAME for user code today. Even if we later
added a worker-side arg, we'd still want the server value to be
authoritative - the DB-resolved bundle mapping is the single source of
truth and can't drift or be spoofed by a mis-passed CLI flag.

Scope of this PR.

Given all of the above, this PR is only a typing sync: the field is
already in TIRunContext.dag_run and already surfaces in user context,
but DagRunProtocol (the Protocol used to type context["dag_run"])
was last synced before #65617 and lags. Type checkers therefore flag
context["dag_run"].team_name as [attr-defined] today. This 8-line
change closes that gap; no runtime behaviour changes.

If you'd rather see team_name moved off DagRun server-side (e.g.
promoted to DagResponse / DagResult), I'm happy to open a follow-up
for that architectural discussion - but it's a separate conversation from
keeping the SDK protocol aligned with what the server already ships.

@dkranchii

Copy link
Copy Markdown
Contributor Author

@ashb can you review this PR. thanks.

@o-nikolas

Copy link
Copy Markdown
Contributor

Quite a good write up from @dkranchii but as the person who added that code to the exec api in the first place @ash the tl;dr is that we needed a reliable way to know the team of a task for triggers, xcom, etc. This is directly from the "server" side (can't be manipulated or mis configured by users.) and used in multiple places. This PR is just updating typing, no need to put any major blockers on this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:task-sdk backport-to-v3-3-test Backport to v3-3-test ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants