Skip to content

Add per-component image overrides to Helm chart #68784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
biankabekefi-cldr wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add per-component image overrides to Helm chart #68784

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions chart/docs/production-guide.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,39 @@ See :ref:`Extending Airflow Image <quick-start:extending-airflow-image>` and/or
`Building the image <https://airflow.apache.org/docs/docker-stack/build.html>`_ for more
details on how you can extend, customize and test the modifications of Airflow image.

Per-component image overrides
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You can set a different image for the API server, scheduler, DAG processor, and triggerer
main containers under ``images.airflow``. This is useful when you build component-specific
images (for example slimmer images per role) while keeping a shared default for other
Airflow containers.

Omitted fields in a component block inherit from ``images.airflow`` and then from
``defaultAirflowRepository``, ``defaultAirflowTag``, and ``defaultAirflowDigest``.
When both tag and digest are set for a component, digest takes precedence.

.. code-block:: yaml
:caption: values.yaml

images:
airflow:
repository: my-registry/airflow
tag: "3.1.0"
scheduler:
repository: my-registry/airflow-scheduler
tag: "3.1.0-slim"
apiServer:
tag: "3.1.0-api"

.. code-block:: bash

helm upgrade --install my-release apache-airflow/airflow \
--set images.airflow.scheduler.repository=my-registry/airflow-scheduler \
--set images.airflow.scheduler.tag=3.1.0-slim

See :doc:`parameters-ref` for the full list of ``images.airflow`` parameters.

Managing Dag Files
------------------

Expand Down
22 changes: 22 additions & 0 deletions chart/templates/_helpers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1167,3 +1167,25 @@ Usage:
path: namespace
{{- end }}
{{- end -}}

{{/*
Configure the image for the different components.

Usage:
{{ include "airflow_component_image" (merge (dict "component" "apiServer") .) }}
*/}}
{{- define "airflow_component_image" -}}
{{- $override := index .Values.images.airflow .component | default dict -}}
{{- $repository := $override.repository | default .Values.images.airflow.repository | default .Values.defaultAirflowRepository -}}
{{- $defaultTag := .Values.images.airflow.tag | default .Values.defaultAirflowTag -}}
{{- $defaultDigest := .Values.images.airflow.digest | default .Values.defaultAirflowDigest -}}
{{- if $override.digest }}
{{- printf "%s@%s" $repository $override.digest -}}
{{- else if $override.tag }}
{{- printf "%s:%s" $repository $override.tag -}}
{{- else if $defaultDigest }}
{{- printf "%s@%s" $repository $defaultDigest -}}
{{- else }}
{{- printf "%s:%s" $repository $defaultTag -}}
{{- end }}
{{- end }}
2 changes: 1 addition & 1 deletion chart/templates/api-server/api-server-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ spec:
{{- end }}
containers:
- name: api-server
image: {{ template "airflow_image" . }}
image: {{ include "airflow_component_image" (merge (dict "component" "apiServer") .) }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
securityContext: {{ $containerSecurityContext | nindent 12 }}
{{- if $containerLifecycleHooks }}
Expand Down
2 changes: 1 addition & 1 deletion chart/templates/dag-processor/dag-processor-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ spec:
{{- end }}
containers:
- name: dag-processor
image: {{ template "airflow_image" . }}
image: {{ include "airflow_component_image" (merge (dict "component" "dagProcessor") .) }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
securityContext: {{ $containerSecurityContext | nindent 12 }}
{{- if $containerLifecycleHooks }}
Expand Down
2 changes: 1 addition & 1 deletion chart/templates/scheduler/scheduler-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ spec:
{{- end }}
containers:
- name: scheduler
image: {{ template "airflow_image" . }}
image: {{ include "airflow_component_image" (merge (dict "component" "scheduler") .) }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
securityContext: {{ $containerSecurityContext | nindent 12 }}
{{- if $containerLifecycleHooks }}
Expand Down
2 changes: 1 addition & 1 deletion chart/templates/triggerer/triggerer-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ spec:
{{- end }}
containers:
- name: triggerer
image: {{ template "airflow_image" . }}
image: {{ include "airflow_component_image" (merge (dict "component" "triggerer") .) }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
securityContext: {{ $containerSecurityContext | nindent 12 }}
{{- if $containerLifecycleHooks }}
Expand Down
33 changes: 33 additions & 0 deletions chart/tests/helm_tests/airflow_aux/test_airflow_common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,39 @@ def test_should_use_correct_default_image(self, expected_image, tag, digest):
for doc in docs:
assert expected_image == jmespath.search("spec.template.spec.initContainers[0].image", doc)

@pytest.mark.parametrize(
("component", "component_key", "expected_image", "digest"),
[
("api-server", "apiServer", "apache/airflow-api-server@api-server-digest", "api-server-digest"),
("scheduler", "scheduler", "apache/airflow-scheduler:scheduler-tag", None),
("dag-processor", "dagProcessor", "apache/airflow-dag-processor@test-digest", "test-digest"),
("triggerer", "triggerer", "apache/airflow-triggerer:triggerer-tag", None),
],
)
def test_should_use_correct_component_image(self, component, component_key, expected_image, digest):
docs = render_chart(
values={
"defaultAirflowRepository": "apache/airflow",
"defaultAirflowTag": "default-tag",
"defaultAirflowDigest": "default-digest",
"images": {
"airflow": {
component_key: {
"repository": f"apache/airflow-{component}",
"tag": f"{component}-tag",
"digest": digest,
},
},
},
},
show_only=[
f"templates/{component}/{component}-deployment.yaml",
],
)

for doc in docs:
assert expected_image == jmespath.search("spec.template.spec.containers[0].image", doc)

def test_should_set_correct_helm_hooks_weight(self):
docs = render_chart(
show_only=["templates/secrets/fernetkey-secret.yaml"],
Expand Down
127 changes: 127 additions & 0 deletions chart/values.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -865,6 +865,133 @@
"IfNotPresent"
],
"default": "IfNotPresent"
},
"apiServer": {
"description": "Optional image override for the API server deployment main container. Omitted fields inherit from ``images.airflow`` and then ``defaultAirflowRepository`` / ``defaultAirflowTag`` / ``defaultAirflowDigest``.",
"type": "object",
"additionalProperties": false,
"properties": {
"repository": {
"description": "The image repository for the API server main container. If omitted, inherits from ``images.airflow.repository`` or ``defaultAirflowRepository``.",
"type": [
"string",
"null"
],
"default": null
},
"tag": {
"description": "The image tag for the API server main container. If omitted, inherits from ``images.airflow.tag`` or ``defaultAirflowTag``.",
"type": [
"string",
"null"
],
"default": null
},
"digest": {
"description": "The image digest for the API server main container. If set, it overrides the tag. If omitted, inherits from ``images.airflow.digest`` or ``defaultAirflowDigest``.",
"type": [
"string",
"null"
],
"default": null
}
}
},
"scheduler": {
"description": "Optional image override for the scheduler deployment main container. Omitted fields inherit from ``images.airflow`` and then ``defaultAirflowRepository`` / ``defaultAirflowTag`` / ``defaultAirflowDigest``.",
"type": "object",
"additionalProperties": false,
"properties": {
"repository": {
"description": "The image repository for the scheduler main container. If omitted, inherits from ``images.airflow.repository`` or ``defaultAirflowRepository``.",
"type": [
"string",
"null"
],
"default": null,
"examples": [
"my-registry/airflow-scheduler"
]
},
"tag": {
"description": "The image tag for the scheduler main container. If omitted, inherits from ``images.airflow.tag`` or ``defaultAirflowTag``.",
"type": [
"string",
"null"
],
"default": null
},
"digest": {
"description": "The image digest for the scheduler main container. If set, it overrides the tag. If omitted, inherits from ``images.airflow.digest`` or ``defaultAirflowDigest``.",
"type": [
"string",
"null"
],
"default": null
}
}
},
"triggerer": {
"description": "Optional image override for the triggerer deployment main container. Omitted fields inherit from ``images.airflow`` and then ``defaultAirflowRepository`` / ``defaultAirflowTag`` / ``defaultAirflowDigest``.",
"type": "object",
"additionalProperties": false,
"properties": {
"repository": {
"description": "The image repository for the triggerer main container. If omitted, inherits from ``images.airflow.repository`` or ``defaultAirflowRepository``.",
"type": [
"string",
"null"
],
"default": null
},
"tag": {
"description": "The image tag for the triggerer main container. If omitted, inherits from ``images.airflow.tag`` or ``defaultAirflowTag``.",
"type": [
"string",
"null"
],
"default": null
},
"digest": {
"description": "The image digest for the triggerer main container. If set, it overrides the tag. If omitted, inherits from ``images.airflow.digest`` or ``defaultAirflowDigest``.",
"type": [
"string",
"null"
],
"default": null
}
}
},
"dagProcessor": {
"description": "Optional image override for the DAG processor deployment main container. Omitted fields inherit from ``images.airflow`` and then ``defaultAirflowRepository`` / ``defaultAirflowTag`` / ``defaultAirflowDigest``.",
"type": "object",
"additionalProperties": false,
"properties": {
"repository": {
"description": "The image repository for the DAG processor main container. If omitted, inherits from ``images.airflow.repository`` or ``defaultAirflowRepository``.",
"type": [
"string",
"null"
],
"default": null
},
"tag": {
"description": "The image tag for the DAG processor main container. If omitted, inherits from ``images.airflow.tag`` or ``defaultAirflowTag``.",
"type": [
"string",
"null"
],
"default": null
},
"digest": {
"description": "The image digest for the DAG processor main container. If set, it overrides the tag. If omitted, inherits from ``images.airflow.digest`` or ``defaultAirflowDigest``.",
"type": [
"string",
"null"
],
"default": null
}
}
}
}
},
Expand Down
6 changes: 6 additions & 0 deletions chart/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,12 @@ images:
# Specifying digest takes precedence over tag.
digest: ~
pullPolicy: IfNotPresent
# Optional per-component image (repository, tag, digest). Omitted fields inherit from
# this images.airflow block and defaultAirflowRepository / defaultAirflowTag / defaultAirflowDigest.
apiServer: {}
scheduler: {}
triggerer: {}
dagProcessor: {}
# To avoid images with user code, you can turn this to 'true' and
# all the 'run-airflow-migrations' and 'wait-for-airflow-migrations' jobs/containers
# will use the images from 'defaultAirflowRepository:defaultAirflowTag' values
Expand Down