Skip to content

Chore/sign backend images#8

Open
DhirenMhatre wants to merge 2 commits into
masterfrom
chore/sign-backend-images
Open

Chore/sign backend images#8
DhirenMhatre wants to merge 2 commits into
masterfrom
chore/sign-backend-images

Conversation

@DhirenMhatre
Copy link
Copy Markdown

Description

This PR fixes #

Notes for Reviewers

Signed commits

  • Yes, I signed my commits.

richiejp added 2 commits May 14, 2026 21:05
Close a trust gap where a registry compromise or MITM could silently
replace a backend image: the gallery YAML tells LocalAI which image to
pull, but until now nothing verified the bytes came from our CI.

Consumer (pkg/oci/cosignverify):
- New package using sigstore-go to verify keyless-cosign signatures.
- OCI 1.1 referrers API + new bundle format (no legacy :tag.sig).
- Policy fields: Issuer / IssuerRegex / Identity / IdentityRegex /
  NotBefore. NotBefore is the revocation lever — keyless Fulcio certs
  are ephemeral so revocation is policy-side; advancing not_before in
  the gallery YAML invalidates every signature predating the cutoff.
- TUF trusted root cached process-wide so N backends from one gallery
  do 1 fetch, not N.

Plumbing:
- pkg/downloader: ImageVerifier interface + WithImageVerifier option
  threaded through DownloadFileWithContext. Verification runs between
  oci.GetImage and oci.ExtractOCIImage, with digest pinning via
  pinnedImageRef to close the TOCTOU window. Skips the verifier's HEAD
  when the ref is already digest-pinned.
- core/config: Gallery.Verification YAML block.
- core/gallery: backendDownloadOptions builds the verifier from the
  policy; applied on initial URI, mirrors, and tag fallbacks.
- core/gallery/upgrade: the upgrade path now routes through the same
  options builder. A regression Ginkgo spec pins this contract —
  without it, UpgradeBackend silently bypassed verification.
- core/cli: --require-backend-integrity (LOCALAI_REQUIRE_BACKEND_INTEGRITY)
  escalates missing policy / empty SHA256 from warn to hard-fail.

Producer (.github/workflows/backend_merge.yml):
- id-token: write at job scope (PR-fork-safe via existing event gate).
- sigstore/cosign-installer@v3 pinned to v2.4.1.
- After each docker buildx imagetools create, resolve the manifest
  list digest and run cosign sign --recursive --new-bundle-format
  --registry-referrers-mode=oci-1-1 against repo@digest. --recursive
  signs the index and every per-arch entry, matching how the consumer
  resolves a tag to a platform-specific manifest before verifying.

Rollout: backend/index.yaml has no `verification:` block yet, so this
PR is backward-compatible — installs proceed with a warning until the
gallery is populated. Strict mode is opt-in.

Assisted-by: claude-code:claude-opus-4-7 [Bash] [Edit] [Read] [Write] [WebSearch] [WebFetch]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
…ad of env

The previous implementation re-exported the --require-backend-integrity
CLI flag into LOCALAI_REQUIRE_BACKEND_INTEGRITY via os.Setenv, then
re-read it in core/gallery via os.Getenv. This leaked process state
into the gallery package and made the flag impossible to override
per-call or test without touching the env.

Add RequireBackendIntegrity to ApplicationConfig (with a matching
WithRequireBackendIntegrity AppOption) and thread the bool through
every install/upgrade path: InstallBackend, InstallBackendFromGallery,
UpgradeBackend, InstallModelFromGallery, InstallExternalBackend,
ApplyGalleryFromString/File, startup.InstallModels. Worker subcommands
gain the same env-bound flag on WorkerFlags so distributed-worker
installs honor it consistently with the worker daemon path.

Add a forbidigo lint rule against os.Getenv / os.LookupEnv / os.Environ
to keep the env-leak pattern from creeping back. Existing offenders
(p2p, config loaders, etc.) are baseline-grandfathered by the existing
new-from-merge-base: origin/master setting; targeted path exclusions
cover the legitimate cases — kong CLI entry points, backend
subprocesses, system capability probes, gRPC AUTH_TOKEN inheritance,
test gating env vars.

Assisted-by: claude-code:claude-opus-4-7
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@codity-dm
Copy link
Copy Markdown

codity-dm Bot commented May 21, 2026

Policy Check Failed

✗ 3/3 policy checks failed:

• Need 2 more approval(s) (0/2) — comment LGTM or approve via review
• Missing ticket reference (expected: JIRA-, ENG-, #*)
• 36 code file(s) changed but no test files added


To merge this PR:

  1. Address the failed checks listed above
  2. Ensure branch protection requires the codity/policy-check status

Configure policies in your dashboard

@codity-dm
Copy link
Copy Markdown

codity-dm Bot commented May 21, 2026

PR Summary

What Changed

  • Added keyless cosign signature verification for OCI backend images using Sigstore/cosign with OIDC identity matching.
  • Added --require-backend-integrity flag that rejects backend installs without verification (OCI) or SHA256 (tarball/HTTP).
  • Updated CI to sign backend images with cosign using OCI 1.1 referrers API.

Key Changes by Area

Security & Verification

  • New pkg/oci/cosignverify/ package verifies OCI images via sigstore-go with issuer/identity regex policies and not_before revocation.
  • Gallery YAML supports verification: block with issuer, identity_regex, and not_before fields.

CLI & Config

  • New RequireBackendIntegrity config flag in core/config/application_config.go:78 and worker config.
  • All backend commands (run, backends, models, worker subcommands) updated to pass integrity requirements.

Backend Operations

  • InstallBackendFromGallery, InstallBackend, UpgradeBackend now accept requireIntegrity bool parameter.
  • Atomic upgrade with rollback using tmp/backup directory swap in core/gallery/upgrade.go:235-370.

CI/CD

  • .github/workflows/backend_merge.yml signs images with --recursive, --new-bundle-format, --registry-referrers-mode=oci-1-1.

Files Changed

File Changes Summary
.agents/adding-backends.md Updated with integrity verification notes
.agents/backend-signing.md New documentation on trust model, CI setup, strict mode, revocation playbook
.github/workflows/backend_merge.yml Added cosign signing step with OCI 1.1 referrers
.golangci.yml Added forbidigo rules banning os.Getenv outside CLI, testing.T methods
core/application/startup.go Pass integrity flag through startup flow
core/application/upgrade_checker.go Pass integrity flag to upgrade checks
core/backend/llm.go Updated backend path resolution with integrity
core/cli/backends.go Added --require-backend-integrity flag
core/cli/models.go Added integrity flag support
core/cli/run.go Added integrity flag support
core/cli/worker/worker.go Pass integrity through worker commands
core/cli/worker/worker_backend_common.go Common worker backend logic with integrity
core/cli/worker/worker_llamacpp.go LLaMA.cpp worker with integrity checks
core/cli/worker/worker_mlx_common.go MLX common worker with integrity
core/cli/worker/worker_mlx_distributed.go MLX distributed worker with integrity
core/cli/worker/worker_p2p.go P2P worker with integrity
core/cli/worker/worker_p2p_mlx.go P2P MLX worker with integrity
core/cli/worker/worker_vllm.go vLLM worker with integrity
core/config/application_config.go Added RequireBackendIntegrity field
core/config/gallery.go Added Verification struct with issuer/identity/not_before
core/gallery/backends.go backendDownloadOptions() with OCI verification and SHA256 enforcement
core/gallery/backends_test.go Tests for integrity enforcement
core/gallery/backends_version_test.go Version-specific backend tests
core/gallery/models.go Model install with integrity support
core/gallery/models_test.go Model tests with integrity
core/gallery/upgrade.go Atomic upgrade with integrity checks and rollback
core/gallery/upgrade_test.go Regression test for strict mode rejecting unverified OCI
core/services/galleryop/backends.go InstallExternalBackend with integrity parameter
core/services/galleryop/backends_test.go Gallery op backend tests
core/services/galleryop/managers_local.go Local manager with integrity
core/services/galleryop/models.go Gallery model operations
core/services/worker/config.go Worker config with RequireBackendIntegrity
core/services/worker/install.go Worker install with integrity
core/startup/model_preload.go Preload with integrity checks
core/startup/model_preload_test.go Preload tests
go.mod Added sigstore-go, rekor, TUF, tink dependencies
go.sum Updated checksums
pkg/downloader/pinned_ref_internal_test.go Tests for image reference pinning
pkg/downloader/uri.go Added ImageVerifier interface, WithImageVerifier(), TOCTOU protection
pkg/oci/cosignverify/bundle.go OCI 1.1 referrers API bundle discovery
pkg/oci/cosignverify/cosignverify_suite_test.go Ginkgo test suite
pkg/oci/cosignverify/notbefore_internal_test.go NotBefore revocation tests
pkg/oci/cosignverify/verify.go Core verification logic with sigstore-go
pkg/oci/cosignverify/verify_test.go Unit and live integration tests with 90s timeout

Review Focus Areas

  • TOCTOU protection: Verify pinnedImageRef() in pkg/downloader/uri.go correctly prevents race conditions between manifest fetch and verification.
  • Strict mode enforcement: Check that requireIntegrity=true rejects OCI backends without verification: blocks before any disk writes (see core/gallery/upgrade_test.go:435-470).
  • Worker CLI plumbing: Ensure all worker subcommands consistently pass integrity flags through findBackendPath and findLLamaCPPBackend.

Architecture

Design Decisions

  • OCI 1.1 referrers API used instead of legacy :sha256-<hex>.sig tags. This is deliberate to align with cosign 2.2+ and avoid tag pollution.
  • TUF trusted root cached process-wide to avoid redundant fetches. Tradeoff: memory overhead vs. network calls.
  • NotBefore timestamp as policy-side revocation rather than certificate-level. This handles ephemeral Fulcio certificates without complex CRL/OCSP.
  • Atomic upgrade with tmp/backup swap: writes to temp dir, swaps atomically, keeps backup for rollback. Accepts disk space cost for safety.

Scalability & Extensibility

  • ImageVerifier interface in pkg/downloader/uri.go allows alternative verification schemes beyond cosign.
  • Gallery Verification struct extensible for future policy types (e.g., key-based, not just keyless).

Risks

  • Intentional: RequireBackendIntegrity is opt-in. Existing deployments without SHA256/cosign will break if enabled without preparation.
  • Intentional: Live cosign tests gated by LOCALAI_COSIGN_LIVE=1 to avoid CI flakiness and external dependencies.
  • Unintentional: forbidigo linter may catch legitimate os.Getenv in non-CLI code. Exemptions added for tests, backends, system probe.

Merge Status

NOT MERGEABLE — PR Score 44/100, below threshold (50)

  • [H4] PR quality score (44) is below merge floor (50)
  • [H6] Code quality raw score (23) is below merge floor (40)

Comment on lines +153 to 165
func processRequests(systemState *system.SystemState, modelLoader *model.ModelLoader, enforceScan, automaticallyInstallBackend bool, galleries []config.Gallery, backendGalleries []config.Gallery, requests []galleryModel, requireBackendIntegrity bool) error {
ctx := context.Background()
var err error
for _, r := range requests {
utils.ResetDownloadTimers()
if r.ID == "" {
err = installModelFromRemoteConfig(ctx, systemState, modelLoader, r.GalleryModel, utils.DisplayDownloadFunction, enforceScan, automaticallyInstallBackend, backendGalleries)
err = installModelFromRemoteConfig(ctx, systemState, modelLoader, r.GalleryModel, utils.DisplayDownloadFunction, enforceScan, automaticallyInstallBackend, backendGalleries, requireBackendIntegrity)

} else {
err = gallery.InstallModelFromGallery(
ctx, galleries, backendGalleries, systemState, modelLoader, r.ID, r.GalleryModel, utils.DisplayDownloadFunction, enforceScan, automaticallyInstallBackend)
ctx, galleries, backendGalleries, systemState, modelLoader, r.ID, r.GalleryModel, utils.DisplayDownloadFunction, enforceScan, automaticallyInstallBackend, requireBackendIntegrity)
}
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Functional High

The loop keeps processing later requests after an installation error, so return immediately on failure instead of only returning the last error.

Suggested fix
func processRequests(systemState *system.SystemState, modelLoader *model.ModelLoader, enforceScan, automaticallyInstallBackend bool, galleries []config.Gallery, backendGalleries []config.Gallery, requests []galleryModel, requireBackendIntegrity bool) error {
	ctx := context.Background()
	for _, r := range requests {
		utils.ResetDownloadTimers()
		var err error
		if r.ID == "" {
			err = installModelFromRemoteConfig(ctx, systemState, modelLoader, r.GalleryModel, utils.DisplayDownloadFunction, enforceScan, automaticallyInstallBackend, backendGalleries, requireBackendIntegrity)
		} else {
			err = gallery.InstallModelFromGallery(
				ctx, galleries, backendGalleries, systemState, modelLoader, r.ID, r.GalleryModel, utils.DisplayDownloadFunction, enforceScan, automaticallyInstallBackend, requireBackendIntegrity)
		}
		if err != nil {
			return err
		}
	}
	return nil
}
Prompt for AI assistance

Copy the prompt below and paste it into ChatGPT, Claude, or any LLM:

You are an expert go developer with deep knowledge of security, performance, and best practices.

### Context

File: core/services/galleryop/models.go
Lines: 153-165
Issue Type: functional-high
Severity: high

Issue Description:
The loop keeps processing later requests after an installation error, so return immediately on failure instead of only returning the last error.

Current Code:
func processRequests(systemState *system.SystemState, modelLoader *model.ModelLoader, enforceScan, automaticallyInstallBackend bool, galleries []config.Gallery, backendGalleries []config.Gallery, requests []galleryModel, requireBackendIntegrity bool) error {
	ctx := context.Background()
	var err error
	for _, r := range requests {
		utils.ResetDownloadTimers()
		if r.ID == "" {
			err = installModelFromRemoteConfig(ctx, systemState, modelLoader, r.GalleryModel, utils.DisplayDownloadFunction, enforceScan, automaticallyInstallBackend, backendGalleries, requireBackendIntegrity)

		} else {
			err = gallery.InstallModelFromGallery(
				ctx, galleries, backendGalleries, systemState, modelLoader, r.ID, r.GalleryModel, utils.DisplayDownloadFunction, enforceScan, automaticallyInstallBackend, requireBackendIntegrity)
		}
	}
	return err
}

---

### Instructions

1. Fix the issue described above
2. Maintain the exact indentation and code style from the original
3. Follow go best practices and language-specific idioms
4. Ensure the fix addresses the root cause, not just the symptoms
5. Add brief inline comments explaining the fix if needed

### Constraints

- Do not change functionality beyond fixing the identified issue
- Preserve existing variable names and function signatures unless they are part of the problem
- Ensure the fix is production-ready

---


Like Dislike Create Issue Jira

Comment on lines +141 to +158
# shellcheck disable=SC2086
docker buildx imagetools create $tags \
$(printf 'quay.io/go-skynet/ci-cache@sha256:%s ' *)
# Resolve the manifest-list digest (any tag points at it) so
# cosign can sign by digest. Signing by tag would leave the
# signature orphaned the next time the tag moves.
first_tag=$(jq -cr '
.tags | map(select(startswith("quay.io/"))) | .[0]
' <<< "$DOCKER_METADATA_OUTPUT_JSON")
digest=$(docker buildx imagetools inspect "$first_tag" --format '{{.Manifest.Digest}}')
# --recursive walks the list and signs every per-arch entry
# too — clients that resolve a tag to a platform-specific
# manifest before checking signatures need the per-arch
# signatures, not just the list-level one.
cosign sign --yes --recursive \
--new-bundle-format \
--registry-referrers-mode=oci-1-1 \
"quay.io/go-skynet/local-ai-backends@${digest}"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Robustness Medium

The * glob is used without validating that /tmp/digests contains only sha256 filenames, so reject unexpected names before passing them to imagetools create.

Also reported at: .github/workflows/backend_merge.yml L175–L185

Suggested fix
          shopt -s nullglob
          digests=()
          for f in *; do
            [[ "$f" =~ ^[a-f0-9]{64}$ ]] || { echo "Unexpected digest file: $f"; exit 1; }
            digests+=("quay.io/go-skynet/ci-cache@sha256:$f")
          done
          [ ${#digests[@]} -gt 0 ] || { echo "No digest files found"; exit 1; }
          docker buildx imagetools create $tags "${digests[@]}"
          first_tag=$(jq -cr '
            .tags | map(select(startswith("quay.io/"))) | .[0]
          ' <<< "$DOCKER_METADATA_OUTPUT_JSON")
          digest=$(docker buildx imagetools inspect "$first_tag" --format '{{.Manifest.Digest}}')
          cosign sign --yes --recursive \
            --new-bundle-format \
            --registry-referrers-mode=oci-1-1 \
            "quay.io/go-skynet/local-ai-backends@${digest}"
Prompt for AI assistance

Copy the prompt below and paste it into ChatGPT, Claude, or any LLM:

You are an expert bash developer with deep knowledge of security, performance, and best practices.

### Context

File: .github/workflows/backend_merge.yml
Lines: 141-158
Issue Type: robustness-medium
Severity: medium

Issue Description:
The `*` glob is used without validating that `/tmp/digests` contains only sha256 filenames, so reject unexpected names before passing them to `imagetools create`.

_Also reported at: `.github/workflows/backend_merge.yml` L175–L185_

Current Code:
          # shellcheck disable=SC2086
          docker buildx imagetools create $tags \
            $(printf 'quay.io/go-skynet/ci-cache@sha256:%s ' *)
          # Resolve the manifest-list digest (any tag points at it) so
          # cosign can sign by digest. Signing by tag would leave the
          # signature orphaned the next time the tag moves.
          first_tag=$(jq -cr '
            .tags | map(select(startswith("quay.io/"))) | .[0]
          ' <<< "$DOCKER_METADATA_OUTPUT_JSON")
          digest=$(docker buildx imagetools inspect "$first_tag" --format '{{.Manifest.Digest}}')
          # --recursive walks the list and signs every per-arch entry
          # too — clients that resolve a tag to a platform-specific
          # manifest before checking signatures need the per-arch
          # signatures, not just the list-level one.
          cosign sign --yes --recursive \
            --new-bundle-format \
            --registry-referrers-mode=oci-1-1 \
            "quay.io/go-skynet/local-ai-backends@${digest}"

---

### Instructions

1. Fix the issue described above
2. Maintain the exact indentation and code style from the original
3. Follow bash best practices and language-specific idioms
4. Ensure the fix addresses the root cause, not just the symptoms
5. Add brief inline comments explaining the fix if needed

### Constraints

- Do not change functionality beyond fixing the identified issue
- Preserve existing variable names and function signatures unless they are part of the problem
- Ensure the fix is production-ready

---


Like Dislike Create Issue Jira

Comment on lines +99 to +105
rc, err := layers[0].Uncompressed()
if err != nil {
return nil, fmt.Errorf("opening referrer blob: %w", err)
}
defer func() { _ = rc.Close() }()

data, err := io.ReadAll(rc)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security Medium

The bundle blob is read with io.ReadAll without enforcing desc.Size, so cap the read to the advertised size and reject oversized artifacts.

Suggested fix
	rc, err := layers[0].Uncompressed()
	if err != nil {
		return nil, fmt.Errorf("opening referrer blob: %w", err)
	}
	defer func() { _ = rc.Close() }()

	if desc.Size <= 0 {
		return nil, errors.New("referrer artifact has invalid size")
	}
	data, err := io.ReadAll(io.LimitReader(rc, desc.Size))
	if err != nil {
		return nil, fmt.Errorf("reading referrer blob: %w", err)
	}
Prompt for AI assistance

Copy the prompt below and paste it into ChatGPT, Claude, or any LLM:

You are an expert go developer with deep knowledge of security, performance, and best practices.

### Context

File: pkg/oci/cosignverify/bundle.go
Lines: 99-105
Issue Type: security-medium
Severity: medium

Issue Description:
The bundle blob is read with io.ReadAll without enforcing desc.Size, so cap the read to the advertised size and reject oversized artifacts.

Current Code:
	rc, err := layers[0].Uncompressed()
	if err != nil {
		return nil, fmt.Errorf("opening referrer blob: %w", err)
	}
	defer func() { _ = rc.Close() }()

	data, err := io.ReadAll(rc)

---

### Instructions

1. Fix the issue described above
2. Maintain the exact indentation and code style from the original
3. Follow go best practices and language-specific idioms
4. Ensure the fix addresses the root cause, not just the symptoms
5. Add brief inline comments explaining the fix if needed

### Constraints

- Do not change functionality beyond fixing the identified issue
- Preserve existing variable names and function signatures unless they are part of the problem
- Ensure the fix is production-ready

---


Like Dislike Create Issue Jira

@codity-dm
Copy link
Copy Markdown

codity-dm Bot commented May 21, 2026

Security Scan Summary

Metric Value
Vulnerabilities Critical: 0
Overall Risk Clean
Files Scanned 45

No critical security issues detected

Scan completed in 106.6s

Security scan powered by Codity.ai

@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented May 21, 2026

Greptile Summary

This PR introduces end-to-end keyless cosign signature verification for LocalAI's OCI backend images, covering both the producer side (CI signs each pushed manifest list with --new-bundle-format --recursive) and the consumer side (a new pkg/oci/cosignverify package verifies the Sigstore bundle via the OCI 1.1 referrers API before any bytes reach disk).

  • New cosignverify packageverify.go resolves the image to its manifest digest, discovers the Sigstore bundle from the OCI referrers API (bundle.go), and verifies against a configurable keyless policy. A process-scoped TUF trusted-root cache avoids redundant network fetches when N backends install from the same gallery.
  • Gallery-level verification policyGallery.Verification (opt-in YAML block) carries the cosign policy; backendDownloadOptions applies it uniformly to the primary URI, mirrors, and tag-fallback URIs in both InstallBackend and UpgradeBackend (also fixing a pre-existing gap where UpgradeBackend passed an empty SHA256).
  • --require-backend-integrity flag — opt-in flag that promotes a missing verification policy (OCI) or SHA256 (tarball/HTTP) from a warning to a hard failure; threaded through every CLI entry point, worker supervisor, and startup preload path.

Confidence Score: 3/5

Core security logic is sound but the TUF cache can lock a running server into a permanently-errored state after a single transient failure, silently disabling integrity enforcement without a process restart.

The cosignverify package does the right things: manifest digest pinning before verification, artifact binding to exact bytes in memory, and the opt-in flag keeps existing deployments unaffected. The UpgradeBackend SHA256 fix is a genuine improvement. The main concern is loadTrustedMaterial's sync.Once permanently caching TUF errors: for a long-running server with --require-backend-integrity, one transient TUF fetch failure blocks all future installs for that process lifetime.

pkg/oci/cosignverify/verify.go — the loadTrustedMaterial sync.Once error-caching behaviour and the stale VerifyImage doc comment; pkg/oci/cosignverify/bundle.go — single-bundle-attempt logic in bundleFromOCISignature

Important Files Changed

Filename Overview
pkg/oci/cosignverify/verify.go New cosign verification package using sigstore-go; TUF trusted-material cache permanently stores errors via sync.Once, stale doc comment on VerifyImage describes legacy approach
pkg/oci/cosignverify/bundle.go OCI 1.1 referrers-based Sigstore bundle discovery; only the first parseable bundle is attempted — verification failure has no fallback to remaining bundles
core/gallery/backends.go Adds backendDownloadOptions gating all install/upgrade paths through the gallery verification policy; download options correctly propagated to mirrors and tag fallbacks
core/gallery/upgrade.go UpgradeBackend now passes SHA256 (was empty string before) and cosign downloadOpts; integrity options computed before any disk writes, fixing a pre-existing bypass
pkg/downloader/uri.go Adds ImageVerifier interface, DownloadOption variadic pattern, and TOCTOU-safe pinnedImageRef; verification runs after manifest fetch and before layer extraction
.github/workflows/backend_merge.yml Adds id-token:write permission and cosign signing step; resolves manifest-list digest by tag before signing by digest to prevent orphaned signatures
core/config/gallery.go Adds GalleryVerification struct and Verification field to Gallery; opt-in design keeps existing galleries unaffected
core/config/application_config.go Adds RequireBackendIntegrity flag (default false) with WithRequireBackendIntegrity option; cleanly threaded through all install/upgrade call sites

Sequence Diagram

sequenceDiagram
    participant CLI as CLI / Startup
    participant BD as backendDownloadOptions
    participant DL as pkg/downloader URI
    participant REG as OCI Registry
    participant CV as cosignverify.Verifier
    participant TUF as Sigstore TUF Mirror
    participant REKOR as Rekor Log

    CLI->>BD: InstallBackend(config, requireIntegrity)
    BD->>BD: Check Gallery.Verification
    alt "No verification policy + requireIntegrity=true"
        BD-->>CLI: error (strict integrity)
    else Verification policy present
        BD->>CV: NewVerifier(policy)
        BD-->>DL: WithImageVerifier(verifier)
    end

    DL->>REG: GetImage(url) fetch manifest
    REG-->>DL: img (manifest resolved)
    DL->>DL: img.Digest() pinned ref
    DL->>CV: VerifyImage(pinned at digest)

    CV->>TUF: loadTrustedMaterial (once per process)
    TUF-->>CV: trusted_root.json

    CV->>REG: Referrers(digest) locate Sigstore bundle
    REG-->>CV: OCI 1.1 referrers index
    CV->>REG: fetch bundle artifact layer
    REG-->>CV: Sigstore bundle JSON

    CV->>REKOR: verify inclusion proof (via sigstore-go)
    REKOR-->>CV: verified
    CV->>CV: check Fulcio cert identity (issuer + SAN regex)
    CV->>CV: enforceNotBefore (if set)
    CV-->>DL: nil (verified)

    DL->>REG: ExtractOCIImage(img) layer pull + unpack
    REG-->>DL: layers extracted to disk
Loading

Reviews (1): Last reviewed commit: "refactor(gallery): plumb RequireBackendI..." | Re-trigger Greptile

Comment on lines +139 to +168
func (v *Verifier) loadTrustedMaterial() (root.TrustedMaterialCollection, error) {
key := trustedMaterialCacheKey{URL: v.policy.TUFRootURL, Path: v.policy.TUFCachePath}
val, _ := trustedMaterialCache.LoadOrStore(key, &trustedMaterialEntry{})
entry := val.(*trustedMaterialEntry)
entry.once.Do(func() {
opts := tuf.DefaultOptions()
if v.policy.TUFRootURL != "" {
opts.RepositoryBaseURL = v.policy.TUFRootURL
}
if v.policy.TUFCachePath != "" {
opts.CachePath = v.policy.TUFCachePath
}
client, err := tuf.New(opts)
if err != nil {
entry.err = fmt.Errorf("cosignverify: initialising TUF client: %w", err)
return
}
trustedRootJSON, err := client.GetTarget("trusted_root.json")
if err != nil {
entry.err = fmt.Errorf("cosignverify: fetching trusted_root.json: %w", err)
return
}
tr, err := root.NewTrustedRootFromJSON(trustedRootJSON)
if err != nil {
entry.err = fmt.Errorf("cosignverify: parsing trusted root: %w", err)
return
}
entry.material = root.TrustedMaterialCollection{tr}
})
return entry.material, entry.err
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 TUF cache permanently poisons on first error

loadTrustedMaterial stores a *trustedMaterialEntry in a sync.Map and initialises it exactly once via sync.Once. If tuf.New or client.GetTarget("trusted_root.json") fails (network blip, cold start before DNS resolves, transient Sigstore CDN hiccup), entry.err is set and the sync.Once is marked done. Every subsequent call for the same (TUFRootURL, TUFCachePath) key returns that same cached error forever for the life of the process, with no way to recover short of a restart.

For a long-running server that opts in to --require-backend-integrity, a single transient TUF failure at the moment of the first backend install permanently disables all future installs until the process is restarted. Consider storing the entry under the key only after a successful initialisation so a future call can retry.

Comment on lines +171 to +177
// VerifyImage resolves imageRef to its manifest digest, fetches the cosign
// signature attachment (the conventional `:sha256-<hex>.sig` tag), assembles
// a Sigstore bundle from the cosign annotations, and verifies that bundle
// against the configured Policy.
//
// Returns nil on the first signature in the attachment that satisfies the
// policy. Returns an error if none do, or if any part of the fetch fails.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The doc comment still describes the legacy cosign approach (:sha256-<hex>.sig tag + per-annotation assembly) rather than the OCI 1.1 referrers / new-bundle-format path that this implementation actually uses.

Suggested change
// VerifyImage resolves imageRef to its manifest digest, fetches the cosign
// signature attachment (the conventional `:sha256-<hex>.sig` tag), assembles
// a Sigstore bundle from the cosign annotations, and verifies that bundle
// against the configured Policy.
//
// Returns nil on the first signature in the attachment that satisfies the
// policy. Returns an error if none do, or if any part of the fetch fails.
// VerifyImage resolves imageRef to its manifest digest, discovers the
// cosign Sigstore bundle via the OCI 1.1 referrers API (new-bundle-format),
// and verifies that bundle against the configured Policy using sigstore-go.
//
// The image ref is pinned to a digest by the caller (pkg/downloader) before
// this is invoked, so the verifier skips a second HEAD and binds the bundle
// directly to the manifest bytes already in memory.
//
// Returns nil when the bundle satisfies the policy (identity, transparency
// log, SCT, and optional NotBefore). Returns an error otherwise.

Comment on lines +66 to +85

var lastErr error
for _, desc := range manifest.Manifests {
if !isSigstoreBundleArtifactType(string(desc.ArtifactType)) {
continue
}
b, err := fetchBundleFromReferrer(ref, desc, opts)
if err != nil {
lastErr = err
continue
}
return b, nil
}
if lastErr != nil {
return nil, fmt.Errorf("cosignverify: no usable Sigstore bundle referrer for %s: %w", digestRef.Name(), lastErr)
}
return nil, fmt.Errorf("cosignverify: no Sigstore bundle referrer for %s (signed with --new-bundle-format?)", digestRef.Name())
}

func fetchBundleFromReferrer(ref name.Reference, desc v1.Descriptor, opts []remote.Option) (*bundle.Bundle, error) {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 First parseable bundle wins; verification failure is not retried against remaining bundles

bundleFromOCISignature returns the first referrer whose JSON parses successfully and hands it to sev.Verify in VerifyImage. If that bundle fails identity verification, VerifyImage returns an error immediately without cycling through any remaining Sigstore bundles in the referrers list. In the current CI setup this is fine, but a re-signed image with multiple bundles could fail even if a valid bundle exists later in the referrers index.

@codity-dm
Copy link
Copy Markdown

codity-dm Bot commented May 21, 2026

Dependency vulnerability scanning

Metric Value
Vulnerabilities Found 1
Scanner npm audit
View vulnerability details (1 items)

1. brace-expansion various

CVE: GHSA-f886-m6hf-6m8v
Severity: MODERATE
Fixed in: True

brace-expansion: Zero-step sequence causes process hang and memory exhaustion

Powered by Codity.ai · Docs

@codity-dm
Copy link
Copy Markdown

codity-dm Bot commented May 21, 2026

Dependency vulnerability scanning

Metric Value
Vulnerabilities Found 4
Scanner pip-audit
View vulnerability details (4 items)

1. pip 24.0

CVE: GHSA-4xh5-x5gv-qwph
Fixed in: 25.3

When extracting a tar archive pip may not check symbolic links point into the extraction directory if the tarfile module doesn't implement PEP 706. Note that upgrading pip to a "fixed" version for thi


2. pip 24.0

CVE: GHSA-6vgw-5pg2-w6jp
Fixed in: 26.0

When pip is installing and extracting a maliciously crafted wheel archive, files may be extracted outside the installation directory. The path traversal is limited to prefixes of the installation dire


3. pip 24.0

CVE: GHSA-58qw-9mgm-455v
Fixed in: 26.1

pip handles concatenated tar and ZIP files as ZIP files regardless of filename or whether a file is both a tar and ZIP file. This behavior could result in confusing installation behavior, such as inst


4. pip 24.0

CVE: GHSA-jp4c-xjxw-mgf9
Fixed in: 26.1

pip prior to version 26.1 would run self-update check functionality after installing wheel files which required importing well-known Python modules names. These module imports were intentionally defer

Powered by Codity.ai · Docs

@codity-dm
Copy link
Copy Markdown

codity-dm Bot commented May 21, 2026

License Compliance Scan

Metric Value
Packages Scanned 980
High Risk (Strong Copyleft) 0
Medium Risk (Weak Copyleft) 6
Low Risk (Permissive) 819
Unknown License 155

Weak copyleft licenses found - verify compatibility

Some packages have unknown licenses - manual review required

Medium Risk Licenses - 6 packages

(MPL-2.0 OR Apache-2.0) (1 packages):

  • dompurify 3.4.0

MPL-2.0 (5 packages):

  • github.com/philippgille/chromem-go 0.7.0
  • github.com/libp2p/go-yamux/v5 5.1.0
  • github.com/hashicorp/golang-lru 1.0.2
  • github.com/hashicorp/golang-lru/v2 2.0.7
  • github.com/shoenig/go-m1cpu 0.1.6
Unknown Licenses - 155 packages
  • jackspeak 3.4.3 (BlueOak-1.0.0)
  • minipass 7.1.3 (BlueOak-1.0.0)
  • package-json-from-dist 1.0.1 (BlueOak-1.0.0)
  • path-scurry 1.11.1 (BlueOak-1.0.0)
  • string-width-cjs 4.2.3
  • strip-ansi-cjs 6.0.1
  • wrap-ansi-cjs 7.0.0
  • github.com/digitorus/timestamp 0.0.0-20231217203849-220c5c2851b7
  • github.com/dunglas/httpsfv 1.1.0
  • github.com/go-jose/go-jose/v4 4.1.4
  • github.com/go-openapi/errors 0.22.4
  • github.com/go-openapi/analysis 0.24.1
  • github.com/go-openapi/runtime 0.29.2
  • github.com/go-openapi/loads 0.23.2
  • github.com/go-openapi/strfmt 0.25.0
  • github.com/go-openapi/swag/stringutils 0.25.4
  • github.com/go-openapi/swag/loading 0.25.4
  • github.com/go-openapi/swag/cmdutils 0.25.4
  • github.com/go-openapi/swag/jsonutils 0.25.4
  • github.com/go-openapi/swag/fileutils 0.25.4

...and 135 more

Powered by Codity.ai · Docs

@codity-dm
Copy link
Copy Markdown

codity-dm Bot commented May 21, 2026

Code Quality Report — test-org-codity/LocalAI · PR #8

Scanned: 2026-05-21 12:55 UTC | Score: 23/100 | Provider: github

Executive Summary

Severity Count
Critical 0
High 0
Medium 3
Low 125
Top Findings

[CQ-LLM-002] .agents/backend-signing.md:1 (Documentation · MEDIUM)

Issue: Missing docstring for the new file.
Suggestion: Add a summary docstring at the top of the file to explain its purpose.

# Backend image signing & verification

[CQ-LLM-003] .github/workflows/backend_merge.yml:64 (Complexity · MEDIUM)

Issue: The addition of multiple nested commands increases complexity.
Suggestion: Consider breaking down the nested commands into smaller, reusable steps.

first_tag=$(jq -cr ' .tags | map(select(startswith("quay.io/"))) | .[0] ' <<< "$DOCKER_METADATA_OUTPUT_JSON")

[CQ-LLM-004] .github/workflows/backend_merge.yml:136 (Error_Handling · MEDIUM)

Issue: Swallowed exception when no tags are found.
Suggestion: Log an error message or handle the case where no tags are found instead of exiting silently.

if [ -z "$tags" ]; then echo "No dockerhub tags from docker/metadata-action; skipping dockerhub merge"; exit 0; fi

[CQ-LLM-001] .agents/adding-backends.md:113 (Documentation · LOW)

Issue: Missing docstring for the new note on integrity.
Suggestion: Add a brief description of the integrity note for clarity.

**Note on integrity:** OCI backends installed from a gallery whose `verification:` block is set are verified against a keyless-cosign policy before extraction; tarball/HTTP backends use the optional `sha256:` field.

[CQ-009] .agents/adding-backends.md:115 (Style · LOW)

Issue: Line exceeds 120 characters (401 chars)
Suggestion: Break long lines into multiple lines for readability

**Note on integrity:** OCI backends installed from a gallery whose `verification:` block is set are verified against a k...

[CQ-008] .agents/backend-signing.md:23 (Maintainability · LOW)

Issue: Magic number 10 in code
Suggestion: Extract to a named constant

- **Revocation:** Keyless cosign certs are ephemeral (10-minute Fulcio

[CQ-009] .agents/backend-signing.md:111 (Style · LOW)

Issue: Line exceeds 120 characters (123 chars)
Suggestion: Break long lines into multiple lines for readability

- `.github/workflows/backend_merge.yml` — producer-side `cosign sign --recursive` after each multi-arch manifest list pu...

[CQ-009] .golangci.yml:58 (Style · LOW)

Issue: Line exceeds 120 characters (263 chars)
Suggestion: Break long lines into multiple lines for readability

          msg: 'Plumb config through ApplicationConfig (or the relevant CLI struct) instead of reading env directly. CLI...

[CQ-009] AGENTS.md:34 (Style · LOW)

Issue: Line exceeds 120 characters (269 chars)
Suggestion: Break long lines into multiple lines for readability

| [.agents/backend-signing.md](.agents/backend-signing.md) | Backend OCI image signing (keyless cosign + sigstore-go) — ...

[CQ-009] core/application/startup.go:215 (Style · LOW)

Issue: Line exceeds 120 characters (317 chars)
Suggestion: Break long lines into multiple lines for readability

	if err := coreStartup.InstallModels(options.Context, application.GalleryService(), options.Galleries, options.BackendGa...

Per-File Breakdown

File Critical High Medium Low Total
.agents/adding-backends.md 0 0 0 2 2
.agents/backend-signing.md 0 0 1 2 3
.github/workflows/backend_merge.yml 0 0 2 0 2
.golangci.yml 0 0 0 1 1
AGENTS.md 0 0 0 1 1
core/application/startup.go 0 0 0 4 4
core/backend/llm.go 0 0 0 1 1
core/cli/backends.go 0 0 0 6 6
core/cli/models.go 0 0 0 2 2
core/cli/run.go 0 0 0 1 1
core/cli/worker/worker.go 0 0 0 5 5
core/cli/worker/worker_backend_common.go 0 0 0 1 1
core/cli/worker/worker_llamacpp.go 0 0 0 1 1
core/cli/worker/worker_mlx_common.go 0 0 0 1 1
core/config/gallery.go 0 0 0 1 1
core/gallery/backends.go 0 0 0 7 7
core/gallery/backends_test.go 0 0 0 3 3
core/gallery/models.go 0 0 0 2 2
core/gallery/models_test.go 0 0 0 1 1
core/gallery/upgrade.go 0 0 0 3 3
core/services/galleryop/backends.go 0 0 0 2 2
core/services/galleryop/managers_local.go 0 0 0 1 1
core/services/galleryop/models.go 0 0 0 9 9
core/services/worker/config.go 0 0 0 5 5
core/services/worker/install.go 0 0 0 2 2
core/startup/model_preload.go 0 0 0 4 4
core/startup/model_preload_test.go 0 0 0 2 2
go.mod 0 0 0 6 6
go.sum 0 0 0 41 41
pkg/downloader/pinned_ref_internal_test.go 0 0 0 2 2
pkg/downloader/uri.go 0 0 0 2 2
pkg/oci/cosignverify/bundle.go 0 0 0 1 1
pkg/oci/cosignverify/notbefore_internal_test.go 0 0 0 1 1
pkg/oci/cosignverify/verify_test.go 0 0 0 2 2

Recommendations

  • Run automated tests after applying fixes to verify no regressions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants