Repository navigation
Airflow Demo: Extend to show 3.x new features #223
Description
Description
Airflow 3.x brings with it significant changes to the architecture (UI, services, auth backends etc.) as well as new features. These (new features) do not figure in our demo and it would be good to showcase some of them e.g.
- run a deferred DAG using the triggerer service
- illustrate using a DAG bundle
- incorporate Opa authorisation
Motivation
- tests things not covered by integration tests
- used as a basis for answering support questions
- motivate users to migrate to 3.x+
Activity
lfrancke commentedon Sep 25, 2025
During refinement decide the scope of the demo and the new features to demonstrate.
adwk67 commentedon Oct 14, 2025
Suggested Scope
See e.g. the 3.0.0 release notes for major changes.
Use the latest product image (3.1.0) for the demo.
1. Run a deferred DAG using the triggerer service
There is an integration test that uses the triggerer, and then can used as a basis for mounting a DAG that makes use of
TimeDeltaTrigger(or similar), illustrating a) delayed DAG runs, and maybe b) DAG runs that persist across a re-start.2. DAG bundles
The documentation is a little thin regarding DAG bundles, but they are basically intended to extend previous the DAGS_FOLDER (allowing DAGs to be fetched from an S3 bucket or a git repo). We already have gitsync integration (via gitsync in our images) so it does not make much sense to use another git mechanism.
3. Validate DAGs with
ruffprior to migration4. OPA authorizer
We also have an integration test and documentation for this, so the demo would not need to do anything "new" but present existing functionality in a more step-by-step manner (also mentioning how the plugin works).
5. Event-driven DAG runs
It is now possible for DAGs to react to events external to Airflow. There is an out-of-the-box example available with 3.0, but it would be nice to Stackable-ize this and show Airflow reacting to e.g. a message being written from Nifi to Kafka.
6. (Optional): HITL workflow
See https://airflow.apache.org/blog/airflow-3.1.0/#-human-in-the-loop-hitl-when-automation-meets-human-judgment. Can't really think of a stackable-specific example for this... (M/L)
lfrancke commentedon Oct 15, 2025
@adwk67 and I looked at the items together and decided on this plan.
Maintaining demos is a lot of work so we've tried focusing on things that highlight our platform and not just demonstrate Airflow features.
adwk67 commentedon Oct 28, 2025
Release notes snippet