Skip to content

fix: run init-job on first install #450

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jonasbadstuebner wants to merge 1 commit into
base: master
Choose a base branch
from
Open

fix: run init-job on first install #450

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 24 additions & 47 deletions cockroachdb/templates/job.init.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,10 @@ metadata:
{{- toYaml . | nindent 4 }}
{{- end }}
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation
{{- if not .Release.IsInstall }}
helm.sh/hook: post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
Comment on lines +23 to +24
Copy link
Copy Markdown
Contributor

@prafull01 prafull01 Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main issue with this is the job will be left in the cluster and when you do an upgrade, the new image will be patched to this job which is not allowed as job is an immutable resource in kubernetes.

Copy link
Copy Markdown
Author

@jonasbadstuebner jonasbadstuebner Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the hook-delete-policy still applies on Upgrade, it removes the old job before creating the one for the upgrade.
Feel free to test it out.

{{- end }}
{{- with .Values.init.jobAnnotations }}
{{- toYaml . | nindent 4 }}
{{- end }}
Expand Down Expand Up @@ -97,57 +99,32 @@ spec:
- name: cluster-init
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
# Run the command in an `while true` loop because this Job is bound
# to come up before the CockroachDB Pods (due to the time needed to
# get PersistentVolumes attached to Nodes), and sleeping 5 seconds
# between attempts is much better than letting the Pod fail when
# the init command does and waiting out Kubernetes' non-configurable
# exponential back-off for Pod restarts.
# Command completes either when cluster initialization succeeds,
# or when cluster has been initialized already.
command:
- /bin/bash
- -c
- >-
- |
{{- if $isClusterInitEnabled }}
initCluster() {
while true; do
Comment on lines -113 to -114
Copy link
Copy Markdown
Contributor

@prafull01 prafull01 Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing this loop can cause issues, what if the init was not successful in first attempt. Job will fail and cockroachdb will not come up.

Copy link
Copy Markdown
Author

@jonasbadstuebner jonasbadstuebner Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cockroach init already runs in a loop. I think this was changed since the Job was created and the Job was never updated..

local output=$(
set -x;

/cockroach/cockroach init \
{{- if .Values.tls.enabled }}
--certs-dir=/cockroach-certs/ \
{{- else }}
--insecure \
{{- end }}
{{- with index .Values.conf "cluster-name" }}
--cluster-name={{.}} \
{{- end }}
--host={{ template "cockroachdb.fullname" . }}-0.{{ template "cockroachdb.fullname" . -}}
:{{ .Values.service.ports.grpc.internal.port | int64 }} \
{{- if .Values.init.pcr.enabled -}}
{{- if .Values.init.pcr.isPrimary }}
--virtualized \
{{- else }}
--virtualized-empty \
{{- end }}
{{- end }}
2>&1);

local exitCode="$?";
echo $output;

if [[ "$output" =~ .*"Cluster successfully initialized".* || "$output" =~ .*"cluster has already been initialized".* ]]; then
break;
fi

echo "Cluster is not ready to be initialized, retrying in 5 seconds"
sleep 5;
done
}

initCluster;
set -x;
/cockroach/cockroach init \
{{- if .Values.tls.enabled }}
--certs-dir=/cockroach-certs/
{{- else }}
--insecure
{{- end }} \
{{- with index .Values.conf "cluster-name" }}
--cluster-name={{.}} \
{{- end }}
--host={{ template "cockroachdb.fullname" . }}-0.{{ template "cockroachdb.fullname" . -}}
:{{ .Values.service.ports.grpc.internal.port | int64 }}
{{- if .Values.init.pcr.enabled -}} \
{{- if .Values.init.pcr.isPrimary }}
--virtualized
{{- else }}
--virtualized-empty
{{- end }}
{{- end }} 2>&1 | tee /tmp/output.txt || grep "ERROR: cluster has already been initialized" /tmp/output.txt
{{- end }}

{{- if $isDatabaseProvisioningEnabled }}
Expand Down