chore: add praktika workflow with review job by maxknv · Pull Request #357 · ClickHouse/clickhouse-operator

maxknv · 2026-10-02T16:08:37Z

Why

Migrate PR and Main CI from GitHub Actions YAML to praktika, and add praktika's code review job.

What

Migrated PR and Main CI from .github/workflows/*.yaml to praktika (ci/workflows/pull_request.py, ci/workflows/main_ci.py); defined and deployed the praktika CI infrastructure.
Added praktika's AI code review job.
Dropped the temporary smart test distribution weighted by test duration — e2e sharding is now round-robin; duration-weighted sharding to be reintroduced later.
Dropped the GitHub Actions test reports (dorny/test-reporter and similar) — everything is in the praktika report now.
Moved the OpenShift compatibility job out to a separate GitHub Actions workflow, as it is not yet migrated — to be addressed later.
Disabled the CodeQL workflow's pull_request trigger (now main + cron only).

Important

The required status check in GitHub branch-protection settings must be switched to the new Ready For Merge [PR] status.

Workflow [PR]

Migrate .github/workflows/docs-lint.yaml (both pull_request and push triggers) to praktika workflows: - Add shared Job.Config registry ci/workflows/job_configs.py (JobConfigs) with the vale, doc-links and api-reference-generated jobs, reused by both Pull Request CI (pull_request.py) and Main CI (main_ci.py). - Bake the lint toolchain (Go, pre-warmed crd-ref-docs, Vale, Node + linkspector) into the runner AMIs via an arch-aware build component in ci/infrastructure/projects.py, so jobs run the existing Makefile targets with no per-job installs. - Drop .github/workflows/docs-lint.yaml; the ci-success-check aggregator is handled natively by praktika. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

clickhouse-operator · 2026-10-02T16:18:27Z

Workflow [PR], commit [a5219d8]

Summary: ✅

Code Review

Result: ❌ Blocking issues

What changed: Migrates pull-request and main CI from GitHub Actions to Praktika, adds supporting infrastructure, caching, structured test reporting, and an AI review job. It also moves OpenShift compatibility to a separate workflow and replaces duration-weighted e2e sharding with deterministic hash-based sharding.

Blocking

Every PR runner receives write access to the shared artifact bucket, including the shared ci_cache namespace. Because cached toolchain archives are later extracted over /, untrusted PR code can poison a predictable cache key and obtain persistent code execution in subsequent CI jobs.

Other issues

The CRD fallback invokes the contents endpoint as POST because gh api -f changes the default method. The resulting 404 can make every baseline look like a new CRD, silently bypassing compatibility checks.
The existing immutable-base-SHA concern also remains: the checker still compares against the moving base branch tip rather than the PR event's pinned base commit, so that thread is reopened.

Investigation: 7/13 rounds, 43 tool calls.

clickhouse-operator · 2026-10-02T16:21:32Z

+            volume_size_gb=100,
+            capacity_reserve=1,
+            image_builder=_IMAGE_BUILDERS_BY_NAME["ci-arm64-image"],
+            ext={"allowed_push_branches": ['NA'], "allowed_pr_base_branches": ['main'], "allowed_users": ['maxknv']},


main_ci.py defines a push workflow for main, but this orchestrator explicitly allows pushes only from the nonexistent NA branch. Consequently, the documentation checks that previously ran after every push to main will never be dispatched. Please include main in allowed_push_branches.

Add the two Go-only ci.yaml jobs to the praktika PR and Main workflows via shared JobConfigs, and remove them from .github/workflows/ci.yaml: - Build and Unit Tests: "go build cmd/main.go && make test-ci" (arm-medium; envtest runs in-process, no Docker). controller-gen/setup-envtest self-install via go-install-tool against the baked Go toolchain. - Fuzz Specs: "make fuzz" (arm-small). Both are change-filtered on Go source paths (the praktika equivalent of ci.yaml's changes non-docs gate). The dorny/test-reporter step is dropped; pass/fail comes from the job exit code. Dropped both jobs from ci.yaml's ci-success-check needs list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tika Add three more ci.yaml jobs to the praktika PR/Main workflows via shared JobConfigs, bake their tooling into the runner image, and remove them from .github/workflows/ci.yaml: - Lint: `make lint` (+ go mod tidy / generate / manifests diff checks). - Helm Test: `make generate-helmchart-ci` + chart diff + helm lint, using the baked helm and kubebuilder. - Check CRD Compatibility: PR-only, advisory (allow_failure). ci/jobs/ check_crd_compat.py fetches the base branch (praktika's ephemeral merge checkout has no base history) and runs `make check-crd-compat`. The crd-breaking-change label gate is a workflow filter hook (ci/jobs/filter_job_hook.py) using praktika.info.Info — no gh calls. Image (ci/infrastructure/projects.py, recipe 1.0.1 -> 1.0.2): new _go_ci_tools_component bakes helm + kubebuilder and pre-warms the Go build/module and pip caches for controller-gen, kustomize, setup-envtest, golangci-lint, actionlint, crd-schema-checker and codespell (versions kept in sync with the Makefile), replacing ci.yaml's marketplace actions + actions/cache. Dropped lint from e2e-test's needs and all three jobs from ci-success-check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Fast-moving Go tooling (Go, helm, kubebuilder, controller-gen, kustomize, setup-envtest, golangci-lint, actionlint, crd-schema-checker, crd-ref-docs, envtest assets) is no longer baked into the runner AMI — version bumps would otherwise force an image rebuild. Instead each Go job runs a pre-hook that provisions the toolchain and caches the whole tree on S3. - ci/jobs/s3_cache.py: generic, project-agnostic S3 filesystem cache (content-addressed tar+zstd bundles, boto3 multipart up/download). Kept self-contained for a later move into praktika. - ci/jobs/go_env.py: pre-hook that parses tool versions from go.mod/Makefile, restores the bundle for that version set, or installs + saves it. Keyed by the version set (not go.sum) so dependency bumps are served incrementally from the warm module cache. - job_configs.py: all Go jobs (build_and_test, fuzz_specs, lint, helm_test, check_crd_compat, api_reference_generated) run the go-env pre-hook. - projects.py: AMI now bakes only the rarely-changing docs tools (Vale, Node/linkspector); removed the Go/helm/kubebuilder component (recipe 1.0.5). - .gitignore / ci/.codespellrc: ignore ci runtime (__pycache__, ci/tmp). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…to-praktika

Replace the single workflow pre-hook (which only prepared the Config job's arch, so amd64 jobs missed) with a prep job per arch that builds the go-env bundle natively and publishes it as a praktika artifact; Go jobs `require` the matching bundle and extract it in their pre-hook. - ci/jobs/go_env.py: add `prepare <path>` (prep job: restore-or-build the bundle tarball, S3-backed) and `install` (consumer pre-hook: extract the required artifact from the input dir, self-provision as fallback). - ci/jobs/s3_cache.py: add download()/upload() for the raw tarball so the prep job and artifact layer share one archive. - job_configs.py: Prepare Go Env (arm/amd) jobs provide go-env-<arch>; every Go job requires its arch's bundle and runs the install pre-hook. - pull_request.py / main_ci.py: register the artifacts, add the prep jobs, drop the old ensure pre-hook. Each arch now builds once per run (in its prep job); same-arch consumers just download+extract. The prep job is praktika-cached by the version-source digest and S3-backed, so it rebuilds only when tool versions / scripts change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Move the Dependabot regenerate reaction out of its dedicated GitHub Actions workflow into the praktika PR workflow as an early job: - ci/jobs/dependabot_regenerate.py: fetch+checkout the real PR head, rerun go mod tidy + make generate/manifests/helmchart/api-ref, and push regenerated files back (App-token push re-triggers the PR run, so no manual dispatch). - filter_job_hook.py: gate it to same-repo dependabot[bot] PRs that touched go.mod/go.sum (the original workflow if: guard). - job_configs.py: Dependabot Regenerate job (go-env arm, gh auth, allow_failure; Lint still guards stale generated files). - pull_request.py: list it first, after the go-env prep jobs. Remove .github/workflows/dependabot-regenerate.yaml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The S3 repo snapshot restores a history-free checkout with no `origin` remote, so the old `git fetch origin` / `git show origin/<base>:...` path failed outright. Obtain the base CRD baselines without local git history: try a shallow base fetch first (legacy clones still use git directly), and fall back to the authenticated GitHub contents API, staging baselines into a temp dir that the Make target reads via the new CRD_BASELINE_DIR. Also return a praktika Result (OK/FAIL/ERROR) carrying the baseline source, base branch, staged counts, and the checker's violations on failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The upgrade variant discovers the release to upgrade from via local git tags (deploy_test.go), so the job pre-fetched them. The S3 repo snapshot restores a history-free checkout with no `origin` remote, so `git fetch --tags origin` failed outright. Resolve the upgrade-from release without local git history: try fetching tags first (legacy clones keep reading them via `git tag --list`), and fall back to the highest vX.Y.Z tag from the GitHub API, exported as UPGRADE_FROM_VERSION (which deploy_test.go already honors). Also wrap setup + the make run so provisioning failures return an ERROR praktika Result instead of an uncaught traceback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…e setup failure A reused runner or a crashed prior job can leave a kind cluster named "kind" behind, and `kind create cluster` refuses to clobber it ("node(s) already exist"). Delete any same-named cluster first in kind_env.create_cluster; the delete is idempotent, so a clean runner is unaffected. This covers both the e2e and compat-e2e jobs. Also wrap e2e.py setup + the make run so provisioning failures return an ERROR praktika Result instead of an uncaught traceback, matching compat_e2e. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

`kind delete cluster` does not always reap node containers that are only stopped (e.g. after disable_containerd_image_store restarts the Docker daemon), so `kind create` still failed with "node(s) already exist" on reused autoscaled runners. Add a backstop that force-removes any container carrying the cluster's kind label (io.x-k8s.kind.cluster=<name>) after the delete. Covers both e2e and compat-e2e. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Replace the Lint job's bare shell command with a driver (ci/jobs/lint.py) that emits a structured praktika Result instead of synthesizing pass/fail from the exit code. Sub-results: - go mod tidy / make generate / make manifests regeneration gates (fail with the offending git diff; tracked changes reverted between gates so one gate's drift does not pollute the next) - golangci-lint: JSON output parsed into a per-issue table (file:line:col [linter] -> message) - codespell, actionlint Also extend _GO_CODE_DIGEST with ./test, ./config and ./.golangci.yml so the Lint job's cache invalidates when the code it lints or its config changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…to-praktika # Conflicts: # .github/workflows/ci.yaml

Main #355 moved the compat-e2e matrix to read test/supported/versions.json (single source of truth, auto-updated). Reproduce that in the praktika compat matrix: _clickhouse_versions() computes latest + supported (highest patch per major.minor, descending) from that file, matching ci.yaml jq, instead of hardcoding versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

golangci-lint wrote both the JSON report and the text summary to stdout, so json.load hit trailing text ("Extra data: line 2"). Send JSON to a file (--output.json.path) and keep text on stdout for the log, parse the file, and attach it to the Result. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Report all golangci-lint issues as a single leaf Result whose info lists every finding (file:line: message (linter)), instead of one sub-result per issue — lint findings aren't independent test cases, so a flat list reads better than a nested table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…despell - sharding.go: resolve gosec G115 (mask hash to 31 bits before int conv), wastedassign (drop error(nil) pre-decls), and wsl_v5 whitespace. - lint driver: write golangci-report.json under ci/tmp so the later codespell pass skips it (the report contains linter names like "decorder" that codespell flagged as typos). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The golangci report (written to ci/tmp) contains linter names codespell reads as typos. The existing `ci/tmp` skip entry is a no-op (codespell matches basenames, not slashed paths), so skip the report by basename in ci/.codespellrc. Also reword the lint.py comment that itself contained the flagged word. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- dependabot_regenerate: stop leaking the contents:write token. Keep the origin URL token-free and authenticate via an http.extraheader set without echoing its value, so neither _run nor git error output can reproduce the credential. - job_configs: add ./tools to _GO_CODE_DIGEST so changes to the Go programs under tools/ (vetted/linted/tested by the Go jobs) no longer skip build_and_test / fuzz / lint via change filtering. - job_configs: set enable_gh_auth=True on check_crd_compat so its gh-api baseline fallback works on S3-snapshot runs instead of ERRORing (gh exits 4 unauthenticated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

clickhouse-operator · 2026-10-07T19:55:08Z

+_PROJECT_S3_PREFIXES = list(
+    dict.fromkeys(
+        [
+            f"{Settings.S3_ARTIFACT_BUCKET}/*",


Blocking: untrusted PR jobs can poison the shared root-extracted cache. This bucket-wide prefix is granted to every runner pool, and the jobs demonstrably need write access for ci_cache. Consequently, code from a fork PR can bypass S3PathCache's client-side write-once logic and directly replace or prepopulate a predictable ci_cache/go-env/<key>.tar.zst. Later jobs download that object and tar -xf it over /, allowing a poisoned archive to replace /usr/local/bin tools and execute in trusted jobs. General PR runners should have read-only cache access and write only to run-specific artifact prefixes; cache publication needs a trusted writer and/or cryptographic verification before extraction.

clickhouse-operator · 2026-10-07T19:55:08Z

+            "gh", "api",
+            "-H", "Accept: application/vnd.github.raw",
+            f"repos/{repo}/contents/{crd}",
+            "-f", f"ref={base}",


gh api switches its default method from GET to POST when -f parameters are supplied. The contents endpoint does not support this POST, and its 404/Not Found response is then classified below as “new CRD”; on snapshot runs this can skip every baseline and report compatibility success without comparing anything. Add --method GET or put the URL-encoded ref in the query string.

maxknv marked this pull request as draft October 2, 2026 16:09

add code review job

8cb11d2

maxknv force-pushed the ci/migrate-docs-lint-to-praktika branch from 6a95ab5 to 8cb11d2 Compare October 2, 2026 16:16

clickhouse-operator Bot reviewed Oct 2, 2026

View reviewed changes

clickhouse-operator Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread ci/workflows/job_configs.py

maxknv force-pushed the ci/migrate-docs-lint-to-praktika branch from 498d14a to 1f4e5d6 Compare October 3, 2026 18:44

maxknv force-pushed the ci/migrate-docs-lint-to-praktika branch from 1f4e5d6 to 9ed673d Compare October 5, 2026 11:49

clickhouse-operator Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread ci/jobs/check_crd_compat.py Outdated

maxknv force-pushed the ci/migrate-docs-lint-to-praktika branch from 1612ed4 to 9be75f5 Compare October 6, 2026 10:28

GrigoryPervakov changed the title ~~Add praktika workflow with review job~~ chore: add praktika workflow with review job Oct 6, 2026

maxknv force-pushed the ci/migrate-docs-lint-to-praktika branch 3 times, most recently from 63413ba to 46bac06 Compare October 6, 2026 14:08

clickhouse-operator Bot reviewed Oct 6, 2026

View reviewed changes

Comment thread .github/workflows/ci.yaml Outdated

maxknv force-pushed the ci/migrate-docs-lint-to-praktika branch from 46bac06 to bc4985e Compare October 6, 2026 18:46

maxknv added 2 commits October 6, 2026 20:53

Merge remote-tracking branch 'origin/main' into ci/migrate-docs-lint-…

7c47e2c

…to-praktika

compat e2e PoC

ba622f4

maxknv force-pushed the ci/migrate-docs-lint-to-praktika branch from 6ebe3d1 to ba622f4 Compare October 7, 2026 07:35

maxknv and others added 2 commits October 7, 2026 10:17

compat e2e full matrix with junit translation

e72acb4

maxknv force-pushed the ci/migrate-docs-lint-to-praktika branch from 865e8f7 to e72acb4 Compare October 7, 2026 08:17

maxknv and others added 4 commits October 7, 2026 11:11

migrate e2e jobs

b27d8d8

migrate bundle job, dependabot post regeneration, disable some yamls

f39c9f8

maxknv and others added 12 commits October 7, 2026 16:09

size vms, enable ai assist

0a0aa82

Merge remote-tracking branch 'origin/main' into ci/migrate-docs-lint-…

8b20115

…to-praktika # Conflicts: # .github/workflows/ci.yaml

ci results skill

7756b8f

fix auth

4909192

maxknv marked this pull request as ready for review October 7, 2026 17:50

maxknv marked this pull request as draft October 7, 2026 17:50

clickhouse-operator Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread ci/jobs/dependabot_regenerate.py

Comment thread ci/workflows/job_configs.py

maxknv and others added 2 commits October 7, 2026 21:33

clickhouse-operator Bot reviewed Oct 7, 2026

View reviewed changes

Navigation Menu

Repository navigation

chore: add praktika workflow with review job - #357

chore: add praktika workflow with review job#357
maxknv wants to merge 27 commits into
mainfrom
ci/migrate-docs-lint-to-praktika

maxknv commented Oct 2, 2026 •

edited

Loading

Uh oh!

clickhouse-operator Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

clickhouse-operator Bot Oct 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clickhouse-operator Bot Oct 7, 2026

Uh oh!

clickhouse-operator Bot Oct 7, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

maxknv commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Uh oh!

clickhouse-operator Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review

Blocking

Other issues

Uh oh!

clickhouse-operator Bot Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clickhouse-operator Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

clickhouse-operator Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

maxknv commented Oct 2, 2026 •

edited

Loading

clickhouse-operator Bot commented Oct 2, 2026 •

edited

Loading