Deprecate Asset.extra and add Asset.event_extra_template

[uranusjr]

Body

Asset.extra was originally designed so users can attach additional useful information to describe an asset. We also designed AssetEvent.extra so when an event is emitted against an asset, it can contain additional data to be used in dags/tasks that are triggered by the event, or read the asset as a data inlet.

However, perhaps due to the very similar name, Asset.extra is constantly being misunderstood as some form of blueprint/template/source to AssetEvent.extra, and key-values defined on the asset should be populated into event extras against the asset.

Combined with the latest attempt to add extra templating (#54885 by @hussein-awala), I feel the confusion is much too significant for this feature. My proposal for the situation is

1. Deprecate Asset.extra to be removed in 4.0. It is not clear whether a key-value store for static additional information is useful. Dags don’t have it, and people don’t seem to be too bothered. We can probably add doc_md etc. if needed.
2. Add Asset.event_extra_template to re-implement Support templated extra in outlets assets #54885. Clearer name, no backward compatibility consideration needed.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[Brunda10]

Hi I would like to work on this issue. Please assign it to me.

[Lee-W]

Deprecate Asset.extra to be removed in 4.0. It is not clear whether a key-value store for static additional information is useful. Dags don’t have it, and people don’t seem to be too bothered. We can probably add doc_md etc. if needed.

I'm thinking of something like Asset.description (str). Can't think of other use cases other than description here 🤔

[kaxil]

If I want to mark an Asset as sensitive/containing PII via attaching some metadata, what would be the recommended way.

[uranusjr]

Hmm. What would you do that for a dag or a task? We can do the same on an asset.

[kaxil]

Hmm. What would you do that for a dag or a task? We can do the same on an asset.

I'd say it is similar it adding a tag on the dag object -- so all dagruns are associated to same tag (via the dag)

[Brunda10]

In DAGs and tasks, we generally use doc_md (or doc) for human-readable descriptions, and tags when we want to classify things (e.g. mark something as pii or sensitive).Ref Link

If we want assets to stay consistent with that model, it sounds reasonable :
• Asset.description – for freeform text, same as doc_md
• Asset.tags – for lightweight classification, similar to DAG/Task tags
• Asset.event_extra_template – for dynamic, event-time metadata

This way assets follow the same model as DAGs and tasks, while also avoiding the confusion around extra.

[hussein-awala]

I agree, this is confusing for me and for many other users.

I saw a PR to revert the PR that added support for rendering the extra. Do you think it would be possible to implement your suggestion before releasing 3.1.0?

[uranusjr]

Let’s do the event_extra_template thing first for 3.1.0; the rest is less important. Feel free to start working on it, I’m going to be working on task sdk for a while before I get to this.