From 7bbf60135af5a62b769cbef55625963543fdeea6 Mon Sep 17 00:00:00 2001 From: Manan JK Date: Mon, 31 Aug 2026 16:30:03 +0530 Subject: [PATCH 1/7] docs: validate multi-artifact model compatibility --- docs/research/multi-artifact-validation.md | 146 ++++++++++++++++++ .../ollama/llava-latest-manifest.json | 36 +++++ 2 files changed, 182 insertions(+) create mode 100644 docs/research/multi-artifact-validation.md create mode 100644 tests/fixtures/ollama/llava-latest-manifest.json diff --git a/docs/research/multi-artifact-validation.md b/docs/research/multi-artifact-validation.md new file mode 100644 index 0000000..e2add03 --- /dev/null +++ b/docs/research/multi-artifact-validation.md @@ -0,0 +1,146 @@ +# Multi-artifact Ollama model validation + +**Date:** 2026-08-31 +**Status:** Phase 0 complete; proceed with a constrained model + projector implementation + +## Decision + +StudioLink can safely support an Ollama manifest containing exactly: + +- one `application/vnd.ollama.image.model` GGUF; and +- zero or one `application/vnd.ollama.image.projector` GGUF. + +Both files must be imported into the same LM Studio `user/repo` directory. The projector alias must start with `mmproj-` and end in `.gguf`. A digest-keyed suffix is accepted by LM Studio, so aliases do not need to retain the original upstream filename. + +The first implementation must reject, rather than partially sync: + +- manifests with more than one model layer; +- manifests with more than one projector layer; +- manifests containing an Ollama adapter layer; +- manifests containing an Ollama draft layer; and +- manifests whose only model representation is an Ollama tensor layer. + +An Ollama draft layer is not part of the importable model bundle. LM Studio treats speculative draft models as separately selected model resources, not as files automatically associated by `lms import`. Phase 1 should report a draft layer as an unsupported semantic extension and block sync instead of claiming exact Ollama-to-LM-Studio parity. + +## Primary-source findings + +### Ollama manifest semantics + +Ollama assigns distinct media types to model, projector, adapter, draft, and tensor data. Its current loader maps model layers to `ModelPath`, projectors to a list of `ProjectorPaths`, adapters to `AdapterPaths`, and a draft layer to `DraftPath`.[^ollama-loader] The current manifest package separately defines tensor and draft media types.[^ollama-layer-types] + +These are semantically active layers, not incidental metadata. Ollama passes adapter and projector path lists into its runner, and represents the draft as a separate model path.[^ollama-runner] Ollama's model-list implementation also adds vision capability when at least one projector layer is present.[^ollama-vision] + +The public `llava:latest` registry manifest retrieved on 2026-08-31 contains one model layer followed by one projector layer, then license, template, and parameter metadata.[^llava-manifest] An exact copy is retained at [`tests/fixtures/ollama/llava-latest-manifest.json`](../../tests/fixtures/ollama/llava-latest-manifest.json). `bakllava:latest` had the same model-plus-projector shape. By contrast, `qwen3-vl:latest` used one unified model layer and no projector, confirming that projector support must be conditional rather than assumed for every vision model. + +### LM Studio import semantics + +The current `lms import` interface accepts one file path per invocation.[^lms-import-source] It constructs the destination as: + +```text +/// +``` + +and preserves that basename for copy, hard-link, and symbolic-link imports.[^lms-target-source] The official CLI documentation likewise describes importing one local model file and provides no batch or artifact-role option.[^lms-import-docs] + +LM Studio's documented speculative-decoding interface selects a separate draft model resource. It does not document automatic association of a bundled draft file.[^lmstudio-draft] + +No current LM Studio documentation or `lms load --help` option exposes an Ollama-style LoRA/adapter association. `lms import` can place an arbitrary file, but placement alone is not evidence that LM Studio can apply an Ollama adapter to its base model. Adapter support is therefore outside the safe initial scope. + +## End-to-end validation + +Validation used: + +- LM Studio `0.4.23+1`; +- `lms` commit `07b7252d6de26a3a58c1bb80ed7e75a2b17f5eb6`; +- the installed Ministral 3 3B GGUF and its projector; and +- temporary hard links, so no model bytes were copied and no model was loaded. + +Temporary aliases were given the planned StudioLink naming shape: + +```text +ministral-3-3b-validation-deadbeef.gguf +mmproj-ministral-3-3b-validation-cafebabe.gguf +``` + +Each alias was imported separately with: + +```text +lms import --yes --user-repo studiolink-validation/mmproj-pair --hard-link +``` + +After both imports, `lms ls --json` returned exactly one inventory entry: + +```json +{ + "type": "llm", + "modelKey": "mmproj-pair", + "publisher": "studiolink-validation", + "path": "studiolink-validation/mmproj-pair/ministral-3-3b-validation-deadbeef.gguf", + "sizeBytes": 2986795328, + "architecture": "mistral3", + "vision": true +} +``` + +The reported size was exactly the base file plus the projector file. The installed LM Studio index identified the `mmproj-...gguf` file as the model's `visionAdapter`. This confirms all of the assumptions needed for one-projector support: + +1. separate `lms import` calls can build one logical model bundle; +2. same-directory placement is sufficient; +3. an `mmproj-` prefix is recognized; +4. digest-keyed renamed files remain associated; and +5. `lms ls --json` is a viable public reconciliation seam for the resulting model. + +### Multiple-projector negative test + +A second reversible experiment imported one model and two `mmproj-*.gguf` files into the same repository. LM Studio returned one vision model but indexed only one projector: + +```json +{ + "entryPoint": "ministral-multi-validation.gguf", + "visionAdapter": "mmproj-ministral-multi-two.gguf", + "allFiles": [ + "ministral-multi-validation.gguf", + "mmproj-ministral-multi-two.gguf" + ] +} +``` + +The other projector was ignored. The public inventory did not explain or expose the ambiguity. StudioLink must therefore reject multiple projectors until LM Studio provides a documented selection mechanism. + +All temporary aliases, imported targets, and resulting inventory entries were removed after validation. + +## Supported-layer matrix + +| Ollama layer | Initial handling | Reason | +|---|---|---| +| `image.model` | Import; exactly one required | Confirmed GGUF entry point | +| `image.projector` | Import when exactly one; `mmproj-` alias | Confirmed as LM Studio `visionAdapter` | +| Second projector | Reject model | LM Studio silently selected only one | +| `image.adapter` | Reject model | Semantically required by Ollama; no documented LM Studio association seam | +| `image.draft` | Reject model | LM Studio selects draft models separately; `lms import` cannot preserve the bundle relationship | +| `image.tensor` | Reject model | Not a standalone GGUF import supported by `lms import` | +| template/system/params/license/config | Do not import as artifacts | Manifest metadata, not LM Studio model files | + +## Consequences for Phase 1 + +1. Add artifacts without changing the user-facing model selection seam. +2. Treat model and projector as required artifacts for readiness and sync. +3. Preserve the existing primary model alias naming; prefix projector aliases with `mmproj-`. +4. Import both into the same `user_repo` using separate CLI calls. +5. Do not allow `--direct` for projector bundles because the opaque blob basename loses the required `mmproj-` role marker. +6. Do not claim support for adapters, drafts, tensor-only manifests, or multiple projectors. +7. Add `lms ls --json` inventory parsing before relying on inferred destination paths for multi-artifact reconciliation. + +## Remaining uncertainty + +The validation covers current GGUF model-plus-projector bundles on LM Studio 0.4.23. LM Studio's projector pairing rule is observable but not documented as a stable interface. StudioLink should retain integration tests around inventory behavior and fail clearly if a future LM Studio version stops reporting the imported bundle as vision-capable. + +[^ollama-loader]: Ollama source, [`server/images.go` lines 717–740](https://github.com/ollama/ollama/blob/f96e7aa0513b9973a0ccc71be414c2ecb9d65b1a/server/images.go#L717-L740). +[^ollama-layer-types]: Ollama source, [`manifest/layer.go` lines 20–24](https://github.com/ollama/ollama/blob/f96e7aa0513b9973a0ccc71be414c2ecb9d65b1a/manifest/layer.go#L20-L24). +[^ollama-runner]: Ollama source, [`server/sched.go` runner construction](https://github.com/ollama/ollama/blob/f96e7aa0513b9973a0ccc71be414c2ecb9d65b1a/server/sched.go#L577) and [`server/routes.go` draft model path](https://github.com/ollama/ollama/blob/f96e7aa0513b9973a0ccc71be414c2ecb9d65b1a/server/routes.go#L2368). +[^ollama-vision]: Ollama source, [`server/model_list_cache.go` lines 341–391](https://github.com/ollama/ollama/blob/f96e7aa0513b9973a0ccc71be414c2ecb9d65b1a/server/model_list_cache.go#L341-L391). +[^llava-manifest]: Ollama registry, [`library/llava:latest` manifest](https://registry.ollama.ai/v2/library/llava/manifests/latest), retrieved 2026-08-31. +[^lms-import-source]: LM Studio CLI source, [single `` argument](https://github.com/lmstudio-ai/lms/blob/07b7252d6de26a3a58c1bb80ed7e75a2b17f5eb6/src/subcommands/importCmd.ts#L71-L74). +[^lms-target-source]: LM Studio CLI source, [destination path construction](https://github.com/lmstudio-ai/lms/blob/07b7252d6de26a3a58c1bb80ed7e75a2b17f5eb6/src/subcommands/importCmd.ts#L176) and [hard-link implementation](https://github.com/lmstudio-ai/lms/blob/07b7252d6de26a3a58c1bb80ed7e75a2b17f5eb6/src/subcommands/importCmd.ts#L247-L252). +[^lms-import-docs]: LM Studio documentation, [`lms import`](https://lmstudio.ai/docs/cli/local-models/import). +[^lmstudio-draft]: LM Studio documentation, [Speculative Decoding](https://lmstudio.ai/docs/app/advanced/speculative-decoding). diff --git a/tests/fixtures/ollama/llava-latest-manifest.json b/tests/fixtures/ollama/llava-latest-manifest.json new file mode 100644 index 0000000..10d97d6 --- /dev/null +++ b/tests/fixtures/ollama/llava-latest-manifest.json @@ -0,0 +1,36 @@ +{ + "schemaVersion": 2, + "mediaType": "application/vnd.docker.distribution.manifest.v2+json", + "config": { + "mediaType": "application/vnd.docker.container.image.v1+json", + "digest": "sha256:7c658f9561e5dbbafb042a00f6a4de57877adddd957809111f3123e272632b4d", + "size": 564 + }, + "layers": [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": "sha256:170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868", + "size": 4108916992 + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": "sha256:72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539", + "size": 624434368 + }, + { + "mediaType": "application/vnd.ollama.image.license", + "digest": "sha256:43070e2d4e532684de521b885f385d0841030efa2b1a20bafb76133a5e1379c1", + "size": 11356 + }, + { + "mediaType": "application/vnd.ollama.image.template", + "digest": "sha256:c43332387573e98fdfad4a606171279955b53d891ba2500552c2984a6560ffb4", + "size": 67 + }, + { + "mediaType": "application/vnd.ollama.image.params", + "digest": "sha256:ed11eda7790d05b49395598a42b155812b17e263214292f7b87d15e14003d337", + "size": 30 + } + ] +} From 1090dbcb80436e3a0a4be4d7bc3db95b8b853acf Mon Sep 17 00:00:00 2001 From: Manan JK Date: Tue, 1 Sep 2026 02:06:20 +0530 Subject: [PATCH 2/7] feat: sync Ollama model-projector bundles --- README.md | 26 +- src/studiolink/__init__.py | 2 +- src/studiolink/cli.py | 19 ++ src/studiolink/lmstudio_adapter.py | 49 +++- src/studiolink/models.py | 227 +++++++++++++++-- src/studiolink/ollama_adapter.py | 132 +++++++--- src/studiolink/ports.py | 10 +- src/studiolink/service.py | 77 +++++- src/studiolink/state.py | 2 +- src/studiolink/syncer.py | 386 +++++++++++++++++++++++------ tests/conftest.py | 35 ++- tests/test_cli.py | 1 + tests/test_lmstudio_adapter.py | 43 ++++ tests/test_models.py | 65 +++++ tests/test_ollama_adapter.py | 105 +++++++- tests/test_service.py | 90 ++++++- tests/test_state.py | 45 +++- tests/test_syncer.py | 264 +++++++++++++++++++- 18 files changed, 1401 insertions(+), 177 deletions(-) diff --git a/README.md b/README.md index 6375dba..cef8326 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ StudioLink is a Python CLI tool that bridges Ollama and LM Studio. It treats Oll ## Features -- 🔍 **Scan** - Discover all GGUF models available in your Ollama library +- 🔍 **Scan** - Discover all GGUF models and supported vision projectors in your Ollama library - 🔄 **Sync** - Import models into LM Studio with one command - 📊 **Status** - Track which models are synced and their current state - 🩺 **Doctor** - Verify prerequisites and diagnose issues @@ -113,7 +113,7 @@ sdl -v sync --all - `--copy` - Use copy mode instead of hard links - `--hard-link` - Force hard link mode (default) - `--symbolic-link` - Use symbolic links -- `--direct` - Import the Ollama blob directly without a human-readable alias +- `--direct` - Import a single-artifact Ollama blob directly without a human-readable alias; model-plus-projector bundles require aliases ### `sdl status` Show discovered models and their sync state. @@ -196,9 +196,9 @@ When scanning or checking status, models can have these states: | State | Description | Action Required | |-------|-------------|-----------------| -| **ready** | Model blob exists and has valid GGUF header | Ready to sync | -| **stale** | Manifest exists but blob is missing | Run `ollama pull ` to restore | -| **invalid** | Blob exists but missing GGUF magic bytes | Model may be corrupted | +| **ready** | Every required model/projector blob exists and has a valid GGUF header | Ready to sync | +| **stale** | A required model/projector blob is missing | Run `ollama pull ` to restore | +| **invalid** | An artifact is malformed or the manifest uses an unsupported artifact combination | Inspect the reported issues | | **synced** | Already imported into LM Studio | No action needed | | **pending** | Ready but not yet synced | Run `sdl sync` | @@ -217,13 +217,13 @@ Default paths are platform-aware; every one can be overridden with an environmen ## How It Works 1. **Scanning** - Reads manifest files from the configured Ollama models directory -2. **Blob Resolution** - Locates GGUF blobs in Ollama's content-addressed blob directory -3. **Validation** - Verifies blobs start with GGUF magic bytes -4. **Import Aliases** - Creates digest-keyed, human-readable `.gguf` aliases. Copy mode uses a temporary alias on Ollama's filesystem so the state and LM Studio directories may live on other volumes -5. **LM Studio Import** - Uses `lms import` CLI to import models -6. **State Tracking** - Saves sync records and the expected LM Studio target to `~/.studiolink/state.json` (written atomically) -7. **Reconciliation** - Status and sync verify that the expected LM Studio target still exists instead of trusting state alone -8. **Pruning** - Pruning is blocked after unavailable or incomplete Ollama scans and preserves aliases required by symbolic imports +2. **Artifact Resolution** - Locates the primary model and optional projector GGUF in Ollama's content-addressed blob directory +3. **Validation** - Verifies every required artifact has a valid digest, exists, and starts with GGUF magic bytes +4. **Import Aliases** - Creates digest-keyed, human-readable `.gguf` aliases; projectors use the LM Studio-recognized `mmproj-` prefix. Copy mode uses temporary aliases on Ollama's filesystem so the state and LM Studio directories may live on other volumes +5. **LM Studio Import** - Uses one `lms import` call per artifact, placing each model/projector pair in the same digest-isolated repository so a re-pull cannot leave LM Studio choosing an older projector +6. **State Tracking** - Saves resumable per-artifact sync records to schema-v2 `~/.studiolink/state.json`; schema-v1 records remain readable +7. **Reconciliation** - Status and sync verify every expected target and confirm projector bundles through `lms ls --json` +8. **Pruning** - Pruning is blocked after unavailable or incomplete Ollama scans and preserves all aliases required by symbolic or partial imports ## Troubleshooting @@ -258,6 +258,8 @@ Run `sdl doctor` to verify: - **No silent fallbacks** - If a hard link is impossible (cross-volume), the sync fails loudly; use `--copy` explicitly - **State persistence** - Tracks imports to avoid redundant operations; state is written atomically and corrupt records are skipped, never crash - **Safe pruning** - Orphaned aliases are cleaned up only after a complete source scan; symbolic-link dependencies are retained +- **Supported bundles** - Any number of Ollama models can be synced; each manifest may contain one primary model GGUF and at most one projector GGUF +- **Explicit limitations** - Manifests with multiple primary models/projectors or Ollama adapter, draft, or tensor artifacts are reported as unsupported rather than partially imported - **CLI-only** - No GUI, designed for automation and scripting ## License diff --git a/src/studiolink/__init__.py b/src/studiolink/__init__.py index 0a93a73..8fe341e 100644 --- a/src/studiolink/__init__.py +++ b/src/studiolink/__init__.py @@ -2,4 +2,4 @@ __all__ = ["__version__"] -__version__ = "0.2.1" +__version__ = "0.3.0" diff --git a/src/studiolink/cli.py b/src/studiolink/cli.py index 4119404..6efc2ab 100644 --- a/src/studiolink/cli.py +++ b/src/studiolink/cli.py @@ -186,6 +186,13 @@ def run_scan(args: argparse.Namespace, service: StudioLinkService) -> int: for model in models: print(f"- {model.canonical_name} [{model.readiness.value}]") print(f" blob: {model.blob_path or 'missing'}") + for artifact in model.artifacts: + if artifact.role.value != "model": + print( + f" {artifact.role.value}: " + f"{artifact.blob_path or 'missing'} " + f"[{artifact.readiness.value}]" + ) if model.issues: print(f" issues: {'; '.join(model.issues)}") return 0 @@ -403,6 +410,18 @@ def _model_to_json(model: object) -> dict[str, object]: "gguf_valid": getattr(model, "gguf_valid"), "issues": list(getattr(model, "issues")), "user_repo": getattr(model, "user_repo"), + "artifacts": [ + { + "role": artifact.role.value, + "media_type": artifact.media_type, + "digest": artifact.digest, + "blob_path": (str(artifact.blob_path) if artifact.blob_path else None), + "readiness": artifact.readiness.value, + "gguf_valid": artifact.gguf_valid, + "issues": list(artifact.issues), + } + for artifact in getattr(model, "artifacts", ()) + ], } diff --git a/src/studiolink/lmstudio_adapter.py b/src/studiolink/lmstudio_adapter.py index 216534f..fe32d02 100644 --- a/src/studiolink/lmstudio_adapter.py +++ b/src/studiolink/lmstudio_adapter.py @@ -1,9 +1,10 @@ from __future__ import annotations +import json import subprocess from studiolink.config import StudioLinkConfig -from studiolink.models import ImportResult, LinkMode +from studiolink.models import ImportResult, LinkMode, LMStudioModel DEFAULT_TIMEOUT = 60 # Importing with --copy moves multi-GB GGUF files; give it an hour instead of @@ -62,6 +63,52 @@ def get_import_capabilities(self) -> set[LinkMode]: capabilities.add(LinkMode.SYMBOLIC_LINK) return capabilities + def list_models(self) -> tuple[LMStudioModel, ...]: + command = [str(self.config.lms_exe), "ls", "--json"] + try: + result = self._run(command, check=True) + except subprocess.CalledProcessError as exc: + raise LMStudioError( + f"Failed to list LM Studio models: {exc.stderr}", + exc.cmd, + exc.returncode, + exc.stderr, + ) from exc + try: + payload = json.loads(result.stdout) + except json.JSONDecodeError as exc: + raise LMStudioError( + f"LM Studio returned invalid model inventory JSON: {exc}", + command, + result.returncode, + result.stderr, + ) from exc + if not isinstance(payload, list): + raise LMStudioError( + "LM Studio model inventory was not a JSON list", + command, + result.returncode, + result.stderr, + ) + inventory: list[LMStudioModel] = [] + for item in payload: + if not isinstance(item, dict): + raise LMStudioError( + "LM Studio model inventory contained an invalid entry", + command, + result.returncode, + result.stderr, + ) + inventory.append( + LMStudioModel( + model_key=str(item.get("modelKey", "")), + path=str(item.get("path", "")), + vision=item.get("vision") is True, + model_type=str(item.get("type", "")), + ) + ) + return tuple(inventory) + def import_model( self, source_path: str, diff --git a/src/studiolink/models.py b/src/studiolink/models.py index f8b5b04..73f065e 100644 --- a/src/studiolink/models.py +++ b/src/studiolink/models.py @@ -31,6 +31,32 @@ class ModelReadiness(StrEnum): INVALID = "invalid" +class ArtifactRole(StrEnum): + MODEL = "model" + PROJECTOR = "projector" + + +@dataclass(slots=True, frozen=True) +class OllamaArtifact: + role: ArtifactRole + media_type: str + digest: str | None + blob_path: Path | None + declared_size: int | None + blob_size: int | None + gguf_valid: bool + manifest_index: int + issues: tuple[str, ...] = () + + @property + def readiness(self) -> ModelReadiness: + if self.gguf_valid: + return ModelReadiness.READY + if any("blob is missing" in issue for issue in self.issues): + return ModelReadiness.STALE + return ModelReadiness.INVALID + + @dataclass(slots=True, frozen=True) class OllamaModel: canonical_name: str @@ -46,11 +72,25 @@ class OllamaModel: blob_size: int | None gguf_valid: bool issues: tuple[str, ...] = () + artifacts: tuple[OllamaArtifact, ...] = () @property def readiness(self) -> ModelReadiness: if self.gguf_valid: return ModelReadiness.READY + if any("not supported" in issue for issue in self.issues): + return ModelReadiness.INVALID + if self.artifacts: + if any( + artifact.readiness is ModelReadiness.INVALID + for artifact in self.artifacts + ): + return ModelReadiness.INVALID + if any( + artifact.readiness is ModelReadiness.STALE + for artifact in self.artifacts + ): + return ModelReadiness.STALE if any( "blob is missing from the Ollama blob store" in issue for issue in self.issues @@ -76,11 +116,30 @@ def user_repo(self) -> str: repo_parts.append(self.repository.replace("/", "--")) return f"ollama/{'-'.join(repo_parts)}" + @property + def import_user_repo(self) -> str: + if not any( + artifact.role is ArtifactRole.PROJECTOR for artifact in self.artifacts + ): + return self.user_repo + identity = "-".join( + (artifact.digest or "unknown").split(":")[-1][:12] + for artifact in self.artifacts + ) + return f"{self.user_repo}--bundle-{identity}" + @property def import_filename(self) -> str: + return self._import_filename(self.model_digest) + + def artifact_import_filename(self, artifact: OllamaArtifact) -> str: + prefix = "mmproj-" if artifact.role is ArtifactRole.PROJECTOR else "" + return prefix + self._import_filename(artifact.digest) + + def _import_filename(self, digest: str | None) -> str: # Keep the hash portion of the digest so re-pulled models (same name, # new digest) get a distinct alias instead of reusing the stale one. - digest_hash = (self.model_digest or "unknown").split(":")[-1][:12] + digest_hash = (digest or "unknown").split(":")[-1][:12] raw = f"{self.canonical_name}-{digest_hash}" safe = "".join( ch if ch.isalnum() or ch in ("-", "_", ".") else "-" for ch in raw @@ -99,26 +158,20 @@ class OllamaScanReport: @dataclass(slots=True, frozen=True) -class SyncRecord: - canonical_name: str +class SyncedArtifact: + role: ArtifactRole digest: str blob_path: Path import_alias_path: Path - user_repo: str - link_mode: LinkMode - imported_at: datetime imported_model_path: Path | None = None import_command: tuple[str, ...] = field(default_factory=tuple) def to_json(self) -> dict[str, object]: return { - "canonical_name": self.canonical_name, + "role": self.role.value, "digest": self.digest, "blob_path": str(self.blob_path), "import_alias_path": str(self.import_alias_path), - "user_repo": self.user_repo, - "link_mode": self.link_mode.value, - "imported_at": self.imported_at.isoformat(), "imported_model_path": ( None if self.imported_model_path is None @@ -128,18 +181,15 @@ def to_json(self) -> dict[str, object]: } @classmethod - def from_json(cls, payload: dict[str, object]) -> "SyncRecord": + def from_json(cls, payload: dict[str, object]) -> "SyncedArtifact": import_command = payload.get("import_command", []) if not isinstance(import_command, list): raise TypeError("import_command must be a list") return cls( - canonical_name=str(payload["canonical_name"]), + role=ArtifactRole(str(payload["role"])), digest=str(payload["digest"]), blob_path=Path(str(payload["blob_path"])), import_alias_path=Path(str(payload["import_alias_path"])), - user_repo=str(payload["user_repo"]), - link_mode=LinkMode(str(payload["link_mode"])), - imported_at=datetime.fromisoformat(str(payload["imported_at"])), imported_model_path=( Path(str(payload["imported_model_path"])) if payload.get("imported_model_path") @@ -149,17 +199,156 @@ def from_json(cls, payload: dict[str, object]) -> "SyncRecord": ) +@dataclass(slots=True, frozen=True) +class SyncRecord: + canonical_name: str + user_repo: str + link_mode: LinkMode + imported_at: datetime + artifacts: tuple[SyncedArtifact, ...] + vision_confirmed: bool = True + + @property + def primary_artifact(self) -> SyncedArtifact: + return next( + artifact + for artifact in self.artifacts + if artifact.role is ArtifactRole.MODEL + ) + + # Compatibility properties for callers and schema-v1 JSON consumers. + @property + def digest(self) -> str: + return self.primary_artifact.digest + + @property + def blob_path(self) -> Path: + return self.primary_artifact.blob_path + + @property + def import_alias_path(self) -> Path: + return self.primary_artifact.import_alias_path + + @property + def imported_model_path(self) -> Path | None: + return self.primary_artifact.imported_model_path + + @property + def import_command(self) -> tuple[str, ...]: + return self.primary_artifact.import_command + + def to_json(self) -> dict[str, object]: + primary = self.primary_artifact + return { + "canonical_name": self.canonical_name, + "user_repo": self.user_repo, + "link_mode": self.link_mode.value, + "imported_at": self.imported_at.isoformat(), + "artifacts": [artifact.to_json() for artifact in self.artifacts], + "vision_confirmed": self.vision_confirmed, + # Keep the original primary-artifact fields additive for CLI JSON + # consumers while state schema v2 adopts the artifacts collection. + "digest": primary.digest, + "blob_path": str(primary.blob_path), + "import_alias_path": str(primary.import_alias_path), + "imported_model_path": ( + None + if primary.imported_model_path is None + else str(primary.imported_model_path) + ), + "import_command": list(primary.import_command), + } + + @classmethod + def from_json(cls, payload: dict[str, object]) -> "SyncRecord": + artifacts_payload = payload.get("artifacts") + if artifacts_payload is not None: + if not isinstance(artifacts_payload, list): + raise TypeError("artifacts must be a list") + artifacts = tuple( + SyncedArtifact.from_json(item) + for item in artifacts_payload + if isinstance(item, dict) + ) + if len(artifacts) != len(artifacts_payload) or not artifacts: + raise TypeError("artifacts must contain artifact objects") + else: + import_command = payload.get("import_command", []) + if not isinstance(import_command, list): + raise TypeError("import_command must be a list") + artifacts = ( + SyncedArtifact( + role=ArtifactRole.MODEL, + digest=str(payload["digest"]), + blob_path=Path(str(payload["blob_path"])), + import_alias_path=Path(str(payload["import_alias_path"])), + imported_model_path=( + Path(str(payload["imported_model_path"])) + if payload.get("imported_model_path") + else None + ), + import_command=tuple(str(item) for item in import_command), + ), + ) + if sum(artifact.role is ArtifactRole.MODEL for artifact in artifacts) != 1: + raise ValueError("sync record must contain exactly one model artifact") + if sum(artifact.role is ArtifactRole.PROJECTOR for artifact in artifacts) > 1: + raise ValueError("sync record cannot contain multiple projector artifacts") + return cls( + canonical_name=str(payload["canonical_name"]), + user_repo=str(payload["user_repo"]), + link_mode=LinkMode(str(payload["link_mode"])), + imported_at=datetime.fromisoformat(str(payload["imported_at"])), + artifacts=artifacts, + vision_confirmed=( + payload.get("vision_confirmed") is True + if any( + artifact.role is ArtifactRole.PROJECTOR for artifact in artifacts + ) + else True + ), + ) + + def is_synced( model: OllamaModel, record: SyncRecord | None, *, expected_target: Path | None = None, ) -> bool: - """Check if state and the imported LM Studio file match the model.""" - if record is None or record.digest != model.model_digest: + """Check if state and every imported LM Studio artifact match the model.""" + if record is None: + return False + expected = ( + tuple((artifact.role, artifact.digest) for artifact in model.artifacts) + if model.artifacts + else ((ArtifactRole.MODEL, model.model_digest),) + ) + actual = tuple((artifact.role, artifact.digest) for artifact in record.artifacts) + if actual != expected: return False - target = record.imported_model_path or expected_target - return target.is_file() if target is not None else True + if any(role is ArtifactRole.PROJECTOR for role, _ in expected): + if not record.vision_confirmed: + return False + for artifact in record.artifacts: + target = artifact.imported_model_path + if artifact.role is ArtifactRole.MODEL and target is None: + target = expected_target + if target is None: + if artifact.role is ArtifactRole.PROJECTOR: + return False + continue + if not target.is_file(): + return False + return True + + +@dataclass(slots=True, frozen=True) +class LMStudioModel: + model_key: str + path: str + vision: bool + model_type: str @dataclass(slots=True, frozen=True) diff --git a/src/studiolink/ollama_adapter.py b/src/studiolink/ollama_adapter.py index 906d891..f5783f8 100644 --- a/src/studiolink/ollama_adapter.py +++ b/src/studiolink/ollama_adapter.py @@ -6,9 +6,20 @@ from pathlib import Path from studiolink.config import StudioLinkConfig -from studiolink.models import OllamaModel, OllamaScanReport +from studiolink.models import ( + ArtifactRole, + OllamaArtifact, + OllamaModel, + OllamaScanReport, +) MODEL_MEDIA_TYPE = "application/vnd.ollama.image.model" +PROJECTOR_MEDIA_TYPE = "application/vnd.ollama.image.projector" +UNSUPPORTED_MEDIA_TYPES = { + "application/vnd.ollama.image.adapter": "adapter", + "application/vnd.ollama.image.draft": "draft", + "application/vnd.ollama.image.tensor": "tensor", +} GGUF_MAGIC = b"GGUF" SHA256_DIGEST_PATTERN = re.compile(r"^sha256:[0-9a-fA-F]{64}$") @@ -119,44 +130,42 @@ def _parse_manifest(self, manifest_path: Path) -> OllamaModel | None: issues.append("manifest layers must be a list") layers = [] - model_layer = None - for layer in layers: - if isinstance(layer, dict) and layer.get("mediaType") == MODEL_MEDIA_TYPE: - model_layer = layer - break - - model_digest = None - blob_path = None - declared_size = None - blob_size = None - gguf_valid = False + artifact_layers: list[tuple[int, dict[str, object], ArtifactRole]] = [] + unsupported_roles: set[str] = set() + for index, layer in enumerate(layers): + if not isinstance(layer, dict): + continue + media_type = layer.get("mediaType") + if media_type == MODEL_MEDIA_TYPE: + artifact_layers.append((index, layer, ArtifactRole.MODEL)) + elif media_type == PROJECTOR_MEDIA_TYPE: + artifact_layers.append((index, layer, ArtifactRole.PROJECTOR)) + elif isinstance(media_type, str) and media_type in UNSUPPORTED_MEDIA_TYPES: + unsupported_roles.add(UNSUPPORTED_MEDIA_TYPES[media_type]) - if model_layer is None: + model_count = sum(role is ArtifactRole.MODEL for _, _, role in artifact_layers) + projector_count = sum( + role is ArtifactRole.PROJECTOR for _, _, role in artifact_layers + ) + if model_count == 0: issues.append("missing Ollama model layer") - else: - model_digest = str(model_layer.get("digest", "")).strip() or None - declared_size = self._as_int(model_layer.get("size")) - if model_digest is None: - issues.append("model layer is missing a digest") - elif SHA256_DIGEST_PATTERN.fullmatch(model_digest) is None: - issues.append("model layer has an invalid SHA-256 digest") - else: - blob_path = self.config.ollama_blobs_dir / model_digest.replace( - ":", "-" - ) - logger.debug("Looking for blob: %s", blob_path) - if not blob_path.exists(): - logger.debug("Blob not found: %s", blob_path) - issues.append("model blob is missing from the Ollama blob store") - else: - blob_size = blob_path.stat().st_size - logger.debug("Found blob: %s (size=%d bytes)", blob_path, blob_size) - if self._has_gguf_header(blob_path): - logger.debug("Blob has valid GGUF header: %s", blob_path) - gguf_valid = True - else: - logger.debug("Blob missing GGUF magic bytes: %s", blob_path) - issues.append("blob does not start with GGUF magic bytes") + elif model_count > 1: + issues.append("multiple Ollama model layers are not supported") + if projector_count > 1: + issues.append("multiple Ollama projector layers are not supported") + for role in sorted(unsupported_roles): + issues.append(f"Ollama {role} layers are not supported") + + artifacts = tuple( + self._parse_artifact(layer, role, index) + for index, layer, role in artifact_layers + ) + for artifact in artifacts: + issues.extend(artifact.issues) + primary = next( + (artifact for artifact in artifacts if artifact.role is ArtifactRole.MODEL), + None, + ) canonical_name = self._canonical_name(registry, namespace, repository, tag) fully_qualified_name = f"{registry}/{namespace}/{repository}:{tag}" @@ -168,11 +177,58 @@ def _parse_manifest(self, manifest_path: Path) -> OllamaModel | None: repository=repository, tag=tag, manifest_path=manifest_path, - model_digest=model_digest, + model_digest=primary.digest if primary else None, + blob_path=primary.blob_path if primary else None, + declared_size=primary.declared_size if primary else None, + blob_size=primary.blob_size if primary else None, + gguf_valid=( + bool(artifacts) + and model_count == 1 + and projector_count <= 1 + and not unsupported_roles + and all(artifact.gguf_valid for artifact in artifacts) + ), + issues=tuple(issues), + artifacts=artifacts, + ) + + def _parse_artifact( + self, layer: dict[str, object], role: ArtifactRole, manifest_index: int + ) -> OllamaArtifact: + label = role.value + digest = str(layer.get("digest", "")).strip() or None + declared_size = self._as_int(layer.get("size")) + blob_path = None + blob_size = None + gguf_valid = False + issues: list[str] = [] + if digest is None: + issues.append(f"{label} layer is missing a digest") + elif SHA256_DIGEST_PATTERN.fullmatch(digest) is None: + issues.append(f"{label} layer has an invalid SHA-256 digest") + else: + blob_path = self.config.ollama_blobs_dir / digest.replace(":", "-") + logger.debug("Looking for %s blob: %s", label, blob_path) + if not blob_path.exists(): + issues.append(f"{label} blob is missing from the Ollama blob store") + else: + blob_size = blob_path.stat().st_size + gguf_valid = self._has_gguf_header(blob_path) + if not gguf_valid: + issues.append( + "blob does not start with GGUF magic bytes" + if role is ArtifactRole.MODEL + else f"{label} blob does not start with GGUF magic bytes" + ) + return OllamaArtifact( + role=role, + media_type=str(layer.get("mediaType", "")), + digest=digest, blob_path=blob_path, declared_size=declared_size, blob_size=blob_size, gguf_valid=gguf_valid, + manifest_index=manifest_index, issues=tuple(issues), ) diff --git a/src/studiolink/ports.py b/src/studiolink/ports.py index 1704185..6272d9c 100644 --- a/src/studiolink/ports.py +++ b/src/studiolink/ports.py @@ -2,7 +2,13 @@ from typing import Protocol, runtime_checkable -from studiolink.models import ImportResult, LinkMode, OllamaModel, OllamaScanReport +from studiolink.models import ( + ImportResult, + LinkMode, + LMStudioModel, + OllamaModel, + OllamaScanReport, +) @runtime_checkable @@ -18,6 +24,8 @@ def get_version(self) -> str | None: ... def get_import_capabilities(self) -> set[LinkMode]: ... + def list_models(self) -> tuple[LMStudioModel, ...]: ... + def import_model( self, source_path: str, diff --git a/src/studiolink/service.py b/src/studiolink/service.py index 55fd20d..b736c48 100644 --- a/src/studiolink/service.py +++ b/src/studiolink/service.py @@ -3,14 +3,17 @@ import logging import os from dataclasses import dataclass +from pathlib import Path from typing import TYPE_CHECKING from studiolink.config import StudioLinkConfig from studiolink.lmstudio_adapter import LMStudioAdapter from studiolink.models import ( + ArtifactRole, DoctorCheck, ImportMode, LinkMode, + LMStudioModel, ModelReadiness, OllamaModel, OllamaScanReport, @@ -74,10 +77,23 @@ def scan_report(self) -> OllamaScanReport: def status(self) -> list[StatusEntry]: models = self.scan() records = self.state.get_all_records() + needs_inventory = any( + model.readiness is ModelReadiness.READY + and any( + artifact.role is ArtifactRole.PROJECTOR for artifact in model.artifacts + ) + and records.get(model.canonical_name) is not None + for model in models + ) + inventory = self.lmstudio.list_models() if needs_inventory else None return [ StatusEntry( model=model, - synced=self._record_is_synced(model, records.get(model.canonical_name)), + synced=self._record_is_synced( + model, + records.get(model.canonical_name), + inventory=inventory, + ), sync_record=records.get(model.canonical_name), ) for model in models @@ -130,11 +146,20 @@ def prune(self, *, dry_run: bool = False) -> PruneReport: raise RuntimeError(f"Ollama manifest scan was incomplete: {details}") models = list(scan.models) - expected: dict[str, OllamaModel] = { - model.import_filename: model - for model in models - if model.readiness is ModelReadiness.READY and model.blob_path is not None - } + expected: dict[str, Path] = {} + for model in models: + if model.readiness is not ModelReadiness.READY: + continue + if model.artifacts: + expected.update( + { + model.artifact_import_filename(artifact): artifact.blob_path + for artifact in model.artifacts + if artifact.blob_path is not None + } + ) + elif model.blob_path is not None: + expected[model.import_filename] = model.blob_path records = self.state.get_all_records() unready_names = { model.canonical_name @@ -142,9 +167,10 @@ def prune(self, *, dry_run: bool = False) -> PruneReport: if model.readiness is not ModelReadiness.READY } protected_aliases = { - record.import_alias_path + artifact.import_alias_path for name, record in records.items() if record.link_mode is LinkMode.SYMBOLIC_LINK or name in unready_names + for artifact in record.artifacts } alias_results: list[PruneResult] = [] @@ -156,12 +182,12 @@ def prune(self, *, dry_run: bool = False) -> PruneReport: if path in protected_aliases: logger.debug("Preserving required import alias: %s", path) continue - model = expected.get(path.name) - if model is not None and same_file(path, model.blob_path or path): + expected_blob = expected.get(path.name) + if expected_blob is not None and same_file(path, expected_blob): continue reason = ( "alias no longer points at the current model blob" - if model is not None + if expected_blob is not None else "model no longer present in Ollama (or was re-pulled)" ) try: @@ -347,7 +373,13 @@ def _volume_compatibility_check(self) -> DoctorCheck: ) return DoctorCheck("hard-link volume compatibility", ok, details) - def _record_is_synced(self, model: OllamaModel, record: SyncRecord | None) -> bool: + def _record_is_synced( + self, + model: OllamaModel, + record: SyncRecord | None, + *, + inventory: tuple[LMStudioModel, ...] | None = None, + ) -> bool: expected_target = None if record is not None and record.imported_model_path is None: expected_target = ( @@ -355,7 +387,28 @@ def _record_is_synced(self, model: OllamaModel, record: SyncRecord | None) -> bo / record.user_repo / record.import_alias_path.name ) - return is_synced(model, record, expected_target=expected_target) + if not is_synced(model, record, expected_target=expected_target): + return False + if not any( + artifact.role is ArtifactRole.PROJECTOR for artifact in model.artifacts + ): + return True + if record is None or inventory is None: + return False + primary_target = record.imported_model_path or expected_target + if primary_target is None: + return False + try: + relative_target = primary_target.relative_to( + self.config.lmstudio_models_dir + ).as_posix() + except ValueError: + return False + return any( + item.path.replace("\\", "/").casefold() == relative_target.casefold() + and item.vision + for item in inventory + ) def _select_models( self, discovered: list[OllamaModel], requested_names: list[str] diff --git a/src/studiolink/state.py b/src/studiolink/state.py index 6a91922..3e661f4 100644 --- a/src/studiolink/state.py +++ b/src/studiolink/state.py @@ -63,7 +63,7 @@ def _parse_record(name: str, record_data: object) -> SyncRecord | None: def _save_raw(self, records: Mapping[str, object]) -> None: """Persist raw JSON dict to disk atomically.""" self.path.parent.mkdir(parents=True, exist_ok=True) - payload = json.dumps({"schema_version": 1, "sync_records": records}, indent=2) + payload = json.dumps({"schema_version": 2, "sync_records": records}, indent=2) tmp_path = self.path.parent / (self.path.name + ".tmp") tmp_path.write_text(payload, encoding="utf-8") os.replace(tmp_path, self.path) diff --git a/src/studiolink/syncer.py b/src/studiolink/syncer.py index 4e16a93..03525df 100644 --- a/src/studiolink/syncer.py +++ b/src/studiolink/syncer.py @@ -3,17 +3,21 @@ import logging import os import subprocess +import time from datetime import datetime, timezone from pathlib import Path from studiolink.config import StudioLinkConfig from studiolink.lmstudio_adapter import LMStudioError from studiolink.models import ( + ArtifactRole, ImportMode, ImportResult, LinkMode, ModelReadiness, + OllamaArtifact, OllamaModel, + SyncedArtifact, SyncRecord, SyncResult, is_synced, @@ -23,6 +27,9 @@ logger = logging.getLogger("studiolink") +INVENTORY_ATTEMPTS = 10 +INVENTORY_RETRY_DELAY = 0.5 + def same_file(left: Path, right: Path) -> bool: """Check whether two paths reference the same inode (hard link).""" @@ -94,7 +101,11 @@ def _sync_one( / existing.user_repo / existing.import_alias_path.name ) - if is_synced(model, existing, expected_target=expected_target): + already_synced = is_synced(model, existing, expected_target=expected_target) + has_projector = any( + artifact.role is ArtifactRole.PROJECTOR for artifact in model.artifacts + ) + if already_synced and not has_projector: logger.debug("Model %s is already synced, skipping", model.canonical_name) return SyncResult( model=model, @@ -103,20 +114,16 @@ def _sync_one( record=existing, ) if model.readiness is ModelReadiness.STALE: - logger.debug("Model %s is stale (blob missing)", model.canonical_name) + logger.debug("Model %s has a missing artifact", model.canonical_name) return SyncResult( model=model, status="error", message=( - "model blob is missing from the Ollama blob store; " + "model artifact is missing from the Ollama blob store; " f"run `ollama pull {model.canonical_name}` to restore it" ), ) - if ( - model.readiness is ModelReadiness.INVALID - or model.blob_path is None - or model.model_digest is None - ): + if model.readiness is ModelReadiness.INVALID: logger.debug("Model %s is invalid: %s", model.canonical_name, model.issues) return SyncResult( model=model, @@ -124,89 +131,298 @@ def _sync_one( message="model is not ready for import: " + "; ".join(model.issues or ("unknown error",)), ) - if import_mode is ImportMode.DIRECT: - alias_path = model.blob_path - alias_created = False - logger.debug("Using Ollama blob directly: %s", alias_path) - else: - try: - alias_path, alias_created = self._ensure_import_alias(model, link_mode) - logger.debug( - "Importing model via LM Studio: %s (mode=%s, dry_run=%s)", - alias_path, - link_mode, - dry_run, - ) - except RuntimeError as exc: - logger.debug("Failed to create import alias: %s", exc) + + artifacts = self._model_artifacts(model) + if import_mode is ImportMode.DIRECT and len(artifacts) > 1: + return SyncResult( + model=model, + status="error", + message=( + "--direct is not supported for multi-artifact models; " + "projector imports require an mmproj- alias" + ), + ) + + existing_by_identity = { + (artifact.role, artifact.digest): artifact + for artifact in (existing.artifacts if existing is not None else ()) + } + completed: list[SyncedArtifact] = [] + last_result: ImportResult | None = None + imported_at = ( + existing.imported_at + if existing is not None + else datetime.now(tz=timezone.utc) + ) + + for artifact in artifacts: + if artifact.digest is None or artifact.blob_path is None: return SyncResult( model=model, status="error", - message=str(exc), + message=f"{artifact.role.value} artifact is not importable", + ) + prior = existing_by_identity.get((artifact.role, artifact.digest)) + prior_target = prior.imported_model_path if prior is not None else None + if prior is not None and prior_target is None: + prior_target = ( + self.config.lmstudio_models_dir + / (existing.user_repo if existing else model.import_user_repo) + / prior.import_alias_path.name ) - - import_error: Exception | None = None - import_result: ImportResult | None = None - try: - import_result = self.lmstudio.import_model( - str(alias_path), - user_repo=model.user_repo, - link_mode=link_mode, - dry_run=dry_run, - ) - logger.debug("LM Studio import completed: %s", import_result.return_code) - except (LMStudioError, subprocess.TimeoutExpired) as exc: - # One failing model must not abort the rest of the batch. - logger.debug("LM Studio import failed: %s", exc) - import_error = exc - finally: - should_remove_alias = dry_run or link_mode is LinkMode.COPY if ( - should_remove_alias - and (alias_created or link_mode is LinkMode.COPY) - and alias_path.exists() - and import_mode is not ImportMode.DIRECT + prior is not None + and prior_target is not None + and prior_target.is_file() ): - logger.debug("Cleaning up temporary import alias: %s", alias_path) + completed.append( + SyncedArtifact( + role=prior.role, + digest=prior.digest, + blob_path=prior.blob_path, + import_alias_path=prior.import_alias_path, + imported_model_path=prior_target, + import_command=prior.import_command, + ) + ) + continue + + if import_mode is ImportMode.DIRECT: + alias_path = artifact.blob_path + alias_created = False + else: try: - alias_path.unlink() - alias_path.parent.rmdir() - except OSError as exc: - logger.warning( - "Could not clean up import alias %s: %s", alias_path, exc + alias_path, alias_created = self._ensure_import_alias( + model, artifact, link_mode + ) + except RuntimeError as exc: + return self._partial_result( + model, + records, + completed, + link_mode, + imported_at, + str(exc), + dry_run, ) - if import_error is not None: - return SyncResult(model=model, status="error", message=str(import_error)) - if import_result is None: - return SyncResult( - model=model, - status="error", - message="LM Studio import returned no result", + import_result: ImportResult | None = None + import_error: Exception | None = None + try: + import_result = self.lmstudio.import_model( + str(alias_path), + user_repo=model.import_user_repo, + link_mode=link_mode, + dry_run=dry_run, + ) + except (LMStudioError, subprocess.TimeoutExpired) as exc: + import_error = exc + finally: + self._clean_temporary_alias( + alias_path, + alias_created, + link_mode, + import_mode, + dry_run, + ) + + if import_error is not None or import_result is None: + message = ( + str(import_error) + if import_error is not None + else "LM Studio import returned no result" + ) + return self._partial_result( + model, + records, + completed, + link_mode, + imported_at, + message, + dry_run, + ) + + last_result = import_result + completed.append( + SyncedArtifact( + role=artifact.role, + digest=artifact.digest, + blob_path=artifact.blob_path, + import_alias_path=alias_path, + imported_model_path=( + self.config.lmstudio_models_dir + / model.import_user_repo + / alias_path.name + ), + import_command=import_result.command, + ) ) + if not dry_run: + partial = self._make_record( + model, completed, link_mode, imported_at, vision_confirmed=False + ) + records[model.canonical_name] = partial + self.state.save(records) - record = SyncRecord( - canonical_name=model.canonical_name, - digest=model.model_digest, - blob_path=model.blob_path, - import_alias_path=alias_path, - user_repo=model.user_repo, - link_mode=link_mode, - imported_at=datetime.now(tz=timezone.utc), - imported_model_path=( - self.config.lmstudio_models_dir / model.user_repo / alias_path.name - ), - import_command=import_result.command, + vision_confirmed = True + if len(artifacts) > 1 and not dry_run: + try: + vision_confirmed = self._vision_bundle_is_indexed(model, completed) + except LMStudioError as exc: + return self._partial_result( + model, + records, + completed, + link_mode, + imported_at, + str(exc), + dry_run, + ) + if not vision_confirmed: + return self._partial_result( + model, + records, + completed, + link_mode, + imported_at, + "LM Studio did not index the projector bundle as vision-capable", + dry_run, + ) + + record = self._make_record( + model, completed, link_mode, imported_at, vision_confirmed=vision_confirmed ) + if not dry_run: + records[model.canonical_name] = record + self.state.save(records) + status = "dry-run" if dry_run else "synced" + if already_synced and last_result is None and not dry_run: + status = "skipped" return SyncResult( model=model, - status="dry-run" if dry_run else "synced", - message=_format_import_message(import_result), + status=status, + message=( + _format_import_message(last_result) + if last_result is not None + else "imported artifacts already present" + ), record=record, ) + @staticmethod + def _model_artifacts(model: OllamaModel) -> tuple[OllamaArtifact, ...]: + if model.artifacts: + return model.artifacts + if model.model_digest is None or model.blob_path is None: + return () + return ( + OllamaArtifact( + role=ArtifactRole.MODEL, + media_type="application/vnd.ollama.image.model", + digest=model.model_digest, + blob_path=model.blob_path, + declared_size=model.declared_size, + blob_size=model.blob_size, + gguf_valid=model.gguf_valid, + manifest_index=0, + issues=model.issues, + ), + ) + + def _make_record( + self, + model: OllamaModel, + artifacts: list[SyncedArtifact], + link_mode: LinkMode, + imported_at: datetime, + *, + vision_confirmed: bool, + ) -> SyncRecord: + return SyncRecord( + canonical_name=model.canonical_name, + user_repo=model.import_user_repo, + link_mode=link_mode, + imported_at=imported_at, + artifacts=tuple(artifacts), + vision_confirmed=vision_confirmed, + ) + + def _partial_result( + self, + model: OllamaModel, + records: dict[str, SyncRecord], + completed: list[SyncedArtifact], + link_mode: LinkMode, + imported_at: datetime, + message: str, + dry_run: bool, + ) -> SyncResult: + record = None + if completed: + record = self._make_record( + model, + completed, + link_mode, + imported_at, + vision_confirmed=False, + ) + if not dry_run: + records[model.canonical_name] = record + self.state.save(records) + return SyncResult(model=model, status="error", message=message, record=record) + + def _vision_bundle_is_indexed( + self, model: OllamaModel, artifacts: list[SyncedArtifact] + ) -> bool: + primary = next( + artifact for artifact in artifacts if artifact.role is ArtifactRole.MODEL + ) + if primary.imported_model_path is None: + return False + try: + expected = primary.imported_model_path.relative_to( + self.config.lmstudio_models_dir + ).as_posix() + except ValueError: + return False + for attempt in range(INVENTORY_ATTEMPTS): + if any( + item.path.replace("\\", "/").casefold() == expected.casefold() + and item.vision + for item in self.lmstudio.list_models() + ): + return True + if attempt + 1 < INVENTORY_ATTEMPTS: + time.sleep(INVENTORY_RETRY_DELAY) + return False + + @staticmethod + def _clean_temporary_alias( + alias_path: Path, + alias_created: bool, + link_mode: LinkMode, + import_mode: ImportMode, + dry_run: bool, + ) -> None: + should_remove = dry_run or link_mode is LinkMode.COPY + if not ( + should_remove + and (alias_created or link_mode is LinkMode.COPY) + and alias_path.exists() + and import_mode is not ImportMode.DIRECT + ): + return + try: + alias_path.unlink() + alias_path.parent.rmdir() + except OSError as exc: + logger.warning("Could not clean up import alias %s: %s", alias_path, exc) + def _ensure_import_alias( - self, model: OllamaModel, link_mode: LinkMode + self, + model: OllamaModel, + artifact: OllamaArtifact, + link_mode: LinkMode, ) -> tuple[Path, bool]: staging_dir = self.config.import_staging_dir if link_mode is LinkMode.COPY: @@ -214,15 +430,25 @@ def _ensure_import_alias( # this temporary hard link on the Ollama filesystem so copy mode # still works when StudioLink state lives on another volume. staging_dir = self.config.ollama_models_dir / ".studiolink-imports" - return _ensure_import_alias(model, staging_dir) + return _ensure_import_alias(model, staging_dir, artifact) -def _ensure_import_alias(model: OllamaModel, staging_dir: Path) -> tuple[Path, bool]: - alias_path = staging_dir / model.import_filename - if model.blob_path is None: +def _ensure_import_alias( + model: OllamaModel, + staging_dir: Path, + artifact: OllamaArtifact | None = None, +) -> tuple[Path, bool]: + blob_path = artifact.blob_path if artifact is not None else model.blob_path + alias_name = ( + model.artifact_import_filename(artifact) + if artifact is not None + else model.import_filename + ) + alias_path = staging_dir / alias_name + if blob_path is None: raise RuntimeError(f"No blob path for model {model.canonical_name}") if alias_path.exists(): - if same_file(alias_path, model.blob_path): + if same_file(alias_path, blob_path): logger.debug("Import alias already exists: %s", alias_path) return alias_path, False # The alias survived a re-pull but now points at the old blob content; @@ -231,8 +457,8 @@ def _ensure_import_alias(model: OllamaModel, staging_dir: Path) -> tuple[Path, b alias_path.unlink() staging_dir.mkdir(parents=True, exist_ok=True) try: - os.link(model.blob_path, alias_path) - logger.debug("Created hard link: %s -> %s", alias_path, model.blob_path) + os.link(blob_path, alias_path) + logger.debug("Created hard link: %s -> %s", alias_path, blob_path) return alias_path, True except OSError as exc: raise RuntimeError(f"Failed to create import alias (hard link): {exc}") from exc diff --git a/tests/conftest.py b/tests/conftest.py index 11c5417..b15e2d4 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -17,9 +17,12 @@ from studiolink.config import StudioLinkConfig # noqa: E402 from studiolink.lmstudio_adapter import LMStudioError # noqa: E402 from studiolink.models import ( # noqa: E402 + ArtifactRole, ImportResult, LinkMode, + LMStudioModel, OllamaModel, + SyncedArtifact, SyncRecord, ) @@ -141,12 +144,17 @@ def make_sync_record( ) -> SyncRecord: return SyncRecord( canonical_name=canonical_name, - digest=digest, - blob_path=Path("blob"), - import_alias_path=Path("alias"), user_repo="ollama/llama3", link_mode=LinkMode.HARD_LINK, imported_at=datetime(2026, 1, 1, tzinfo=timezone.utc), + artifacts=( + SyncedArtifact( + role=ArtifactRole.MODEL, + digest=digest, + blob_path=Path("blob"), + import_alias_path=Path("alias"), + ), + ), ) @@ -158,6 +166,7 @@ def __init__(self) -> None: self.fail_for: set[str] = set() self.timeout_for: set[str] = set() self.models_dir: Path | None = None + self.inventory_override: tuple[LMStudioModel, ...] | None = None def get_version(self) -> str | None: return "0.3.17 (fake)" @@ -165,6 +174,26 @@ def get_version(self) -> str | None: def get_import_capabilities(self) -> set[LinkMode]: return {LinkMode.HARD_LINK, LinkMode.COPY, LinkMode.SYMBOLIC_LINK} + def list_models(self) -> tuple[LMStudioModel, ...]: + if self.inventory_override is not None: + return self.inventory_override + if self.models_dir is None or not self.models_dir.exists(): + return () + inventory = [] + for path in self.models_dir.rglob("*.gguf"): + if path.name.startswith("mmproj-"): + continue + relative = path.relative_to(self.models_dir) + inventory.append( + LMStudioModel( + model_key=path.stem, + path=relative.as_posix(), + vision=any(path.parent.glob("mmproj-*.gguf")), + model_type="llm", + ) + ) + return tuple(inventory) + def import_model( self, source_path: str, diff --git a/tests/test_cli.py b/tests/test_cli.py index 6198958..02379a9 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -186,6 +186,7 @@ def test_scan_json(self, fake_service, capsys): payload = json.loads(out(capsys)) assert payload[0]["canonical_name"] == "llama3:1b" assert payload[0]["readiness"] == "ready" + assert payload[0]["artifacts"] == [] def test_status_json(self, fake_service, capsys): assert cli.main(["status", "--json"]) == 0 diff --git a/tests/test_lmstudio_adapter.py b/tests/test_lmstudio_adapter.py index 8b9858c..1153703 100644 --- a/tests/test_lmstudio_adapter.py +++ b/tests/test_lmstudio_adapter.py @@ -1,9 +1,52 @@ +import json +import subprocess + import pytest from studiolink.lmstudio_adapter import LMStudioAdapter, LMStudioError from studiolink.models import LinkMode +def test_list_models_parses_json_inventory(make_config, monkeypatch): + config = make_config() + adapter = LMStudioAdapter(config) + payload = [ + { + "type": "llm", + "modelKey": "mmproj-pair", + "path": "ollama/llava/llava-deadbeef.gguf", + "vision": True, + } + ] + + monkeypatch.setattr( + adapter, + "_run", + lambda *args, **kwargs: subprocess.CompletedProcess( + args[0], 0, json.dumps(payload), "" + ), + ) + + inventory = adapter.list_models() + + assert len(inventory) == 1 + assert inventory[0].model_key == "mmproj-pair" + assert inventory[0].path == "ollama/llava/llava-deadbeef.gguf" + assert inventory[0].vision is True + + +def test_list_models_rejects_invalid_json(make_config, monkeypatch): + adapter = LMStudioAdapter(make_config()) + monkeypatch.setattr( + adapter, + "_run", + lambda *args, **kwargs: subprocess.CompletedProcess(args[0], 0, "nope", ""), + ) + + with pytest.raises(LMStudioError, match="invalid model inventory JSON"): + adapter.list_models() + + def test_import_wraps_os_execution_error(make_config, monkeypatch): config = make_config() adapter = LMStudioAdapter(config) diff --git a/tests/test_models.py b/tests/test_models.py index fe9b8f1..4f9cea0 100644 --- a/tests/test_models.py +++ b/tests/test_models.py @@ -65,6 +65,46 @@ def test_custom_registry_dots_replaced(self): assert model.user_repo == "ollama/ghcr-io-team-gemma" +class TestImportRepository: + def test_projector_bundles_use_digest_isolated_repository(self): + from dataclasses import replace + from pathlib import Path + + from studiolink.models import ArtifactRole, OllamaArtifact + + model = make_ollama_model("llava:latest", DIGEST_A) + projector = OllamaArtifact( + role=ArtifactRole.PROJECTOR, + media_type="application/vnd.ollama.image.projector", + digest=DIGEST_B, + blob_path=Path("projector"), + declared_size=1, + blob_size=1, + gguf_valid=True, + manifest_index=1, + ) + primary = OllamaArtifact( + role=ArtifactRole.MODEL, + media_type="application/vnd.ollama.image.model", + digest=DIGEST_A, + blob_path=Path("model"), + declared_size=1, + blob_size=1, + gguf_valid=True, + manifest_index=0, + ) + model = replace(model, artifacts=(primary, projector)) + + assert model.import_user_repo == ( + "ollama/llava--bundle-aaaaaaaaaaaa-bbbbbbbbbbbb" + ) + changed = replace( + model, + artifacts=(primary, replace(projector, digest="sha256:" + "c" * 64)), + ) + assert changed.import_user_repo != model.import_user_repo + + class TestReadiness: def test_ready(self): assert make_ollama_model("m:1").readiness is ModelReadiness.READY @@ -109,6 +149,31 @@ def test_import_command_defaults_empty(self): restored = make_sync_record().to_json() assert restored["import_command"] == [] + def test_projector_artifact_roundtrip(self): + from dataclasses import replace + from pathlib import Path + + from studiolink.models import ArtifactRole, SyncedArtifact + + record = make_sync_record() + projector = SyncedArtifact( + role=ArtifactRole.PROJECTOR, + digest=DIGEST_B, + blob_path=Path("projector-blob"), + import_alias_path=Path("mmproj-model.gguf"), + imported_model_path=Path("models/ollama/model/mmproj-model.gguf"), + import_command=("lms", "import"), + ) + record = replace( + record, + artifacts=(*record.artifacts, projector), + vision_confirmed=True, + ) + + restored = type(record).from_json(record.to_json()) + + assert restored == record + class TestPruneTypes: def test_prune_result_json(self): diff --git a/tests/test_ollama_adapter.py b/tests/test_ollama_adapter.py index f8a4eb7..e81ef69 100644 --- a/tests/test_ollama_adapter.py +++ b/tests/test_ollama_adapter.py @@ -1,5 +1,7 @@ +import json from pathlib import Path +import pytest from conftest import ( DIGEST_A, blob_filename, @@ -7,7 +9,7 @@ write_manifest, ) -from studiolink.models import ModelReadiness +from studiolink.models import ArtifactRole, ModelReadiness from studiolink.ollama_adapter import OllamaAdapter @@ -32,6 +34,28 @@ def test_finds_ready_model(self, make_config): assert model.declared_size == 1234 assert model.issues == () + def test_scans_model_and_projector_as_one_ready_model(self, make_config): + config = make_config() + fixture = ( + Path(__file__).parent / "fixtures" / "ollama" / "llava-latest-manifest.json" + ) + content = json.loads(fixture.read_text(encoding="utf-8")) + write_manifest(config, repository="llava", content=content) + for layer in content["layers"][:2]: + write_blob(config, layer["digest"]) + + model = scan_one(config) + + assert model.canonical_name == "llava:1b" + assert model.readiness is ModelReadiness.READY + assert [artifact.role for artifact in model.artifacts] == [ + ArtifactRole.MODEL, + ArtifactRole.PROJECTOR, + ] + assert model.artifact_import_filename(model.artifacts[1]).startswith( + "mmproj-llava-1b-" + ) + def test_declared_size_garbage_is_tolerated(self, make_config): config = make_config() write_manifest( @@ -76,6 +100,85 @@ def test_missing_model_layer(self, make_config): assert model.readiness is ModelReadiness.INVALID assert "missing Ollama model layer" in model.issues + def test_missing_projector_blob_makes_bundle_stale(self, make_config): + config = make_config() + write_manifest( + config, + content={ + "layers": [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": DIGEST_A, + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": "sha256:" + "c" * 64, + }, + ] + }, + ) + write_blob(config, DIGEST_A) + + model = scan_one(config) + + assert model.readiness is ModelReadiness.STALE + assert "projector blob is missing from the Ollama blob store" in model.issues + + @pytest.mark.parametrize( + ("extra_layers", "issue"), + [ + ( + [ + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": "sha256:" + "c" * 64, + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": "sha256:" + "d" * 64, + }, + ], + "multiple Ollama projector layers are not supported", + ), + ( + [ + { + "mediaType": "application/vnd.ollama.image.adapter", + "digest": "sha256:" + "c" * 64, + } + ], + "Ollama adapter layers are not supported", + ), + ( + [ + { + "mediaType": "application/vnd.ollama.image.draft", + "digest": "sha256:" + "c" * 64, + } + ], + "Ollama draft layers are not supported", + ), + ], + ) + def test_unsupported_artifact_combinations_are_invalid( + self, make_config, extra_layers, issue + ): + config = make_config() + layers = [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": DIGEST_A, + }, + *extra_layers, + ] + write_manifest(config, content={"layers": layers}) + write_blob(config, DIGEST_A) + + model = scan_one(config) + + assert model.readiness is ModelReadiness.INVALID + assert issue in model.issues + class TestMalformedManifests: def test_io_failure_is_isolated_and_marks_scan_incomplete( diff --git a/tests/test_service.py b/tests/test_service.py index bc49bd7..c4b02ef 100644 --- a/tests/test_service.py +++ b/tests/test_service.py @@ -24,6 +24,13 @@ def build_service(config, lmstudio): ) +def replace_primary(record, **changes): + return replace( + record, + artifacts=(replace(record.primary_artifact, **changes),), + ) + + class TestDoctorVolumeCheck: def test_same_device_passes(self, make_config, fake_lmstudio): config = make_config() @@ -157,7 +164,7 @@ def test_preserves_alias_when_current_manifest_is_not_ready( alias = config.import_staging_dir / model.import_filename os.link(blob, alias) StateStore(config.state_file).upsert( - replace( + replace_primary( make_sync_record(model.canonical_name, DIGEST_A), import_alias_path=alias, ) @@ -182,8 +189,10 @@ def test_preserves_alias_required_by_symbolic_import( alias = config.import_staging_dir / model.import_filename os.link(blob, alias) record = replace( - make_sync_record(model.canonical_name, DIGEST_A), - import_alias_path=alias, + replace_primary( + make_sync_record(model.canonical_name, DIGEST_A), + import_alias_path=alias, + ), link_mode=LinkMode.SYMBOLIC_LINK, ) StateStore(config.state_file).upsert(record) @@ -210,6 +219,44 @@ def test_keeps_healthy_alias(self, make_config, fake_lmstudio): assert report.aliases == () assert (config.import_staging_dir / model.import_filename).exists() + def test_keeps_all_healthy_projector_bundle_aliases( + self, make_config, fake_lmstudio + ): + config = make_config() + fake_lmstudio.models_dir = config.lmstudio_models_dir + service = build_service(config, fake_lmstudio) + projector_digest = "sha256:" + "c" * 64 + write_manifest( + config, + repository="llava", + content={ + "layers": [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": DIGEST_A, + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": projector_digest, + }, + ] + }, + ) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + model = service.scan()[0] + assert service.sync(model_names=[model.canonical_name])[0].status == "synced" + + report = service.prune() + + assert report.aliases == () + assert all( + ( + config.import_staging_dir / model.artifact_import_filename(artifact) + ).exists() + for artifact in model.artifacts + ) + def test_removes_same_name_alias_pointing_elsewhere( self, make_config, fake_lmstudio ): @@ -341,7 +388,7 @@ def test_status_reports_synced_only_when_imported_target_exists( target.parent.mkdir(parents=True) target.write_bytes(b"GGUFxxxx") StateStore(config.state_file).upsert( - replace( + replace_primary( make_sync_record("llama3:1b", DIGEST_A), imported_model_path=target, ) @@ -380,6 +427,39 @@ def test_status_reconciles_legacy_record_with_derived_target( assert entries[0].display_status == "pending" assert entries[0].synced is False + def test_status_requires_projector_bundle_in_lm_studio_inventory( + self, make_config, fake_lmstudio + ): + config = make_config() + fake_lmstudio.models_dir = config.lmstudio_models_dir + service = build_service(config, fake_lmstudio) + projector_digest = "sha256:" + "c" * 64 + write_manifest( + config, + repository="llava", + content={ + "layers": [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": DIGEST_A, + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": projector_digest, + }, + ] + }, + ) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + assert service.sync(model_names=["llava:1b"])[0].status == "synced" + fake_lmstudio.inventory_override = () + + entries = service.status() + + assert entries[0].display_status == "pending" + assert entries[0].synced is False + def test_status_reports_pending_when_imported_target_was_deleted( self, make_config, fake_lmstudio ): @@ -389,7 +469,7 @@ def test_status_reports_pending_when_imported_target_was_deleted( write_blob(config, DIGEST_A) missing_target = config.lmstudio_models_dir / "ollama" / "llama3" / "gone.gguf" StateStore(config.state_file).upsert( - replace( + replace_primary( make_sync_record("llama3:1b", DIGEST_A), imported_model_path=missing_target, ) diff --git a/tests/test_state.py b/tests/test_state.py index 414fa20..826d7db 100644 --- a/tests/test_state.py +++ b/tests/test_state.py @@ -3,7 +3,7 @@ import pytest from conftest import DIGEST_A, make_sync_record -from studiolink.models import SyncRecord +from studiolink.models import ArtifactRole, SyncRecord from studiolink.state import StateStore @@ -79,6 +79,14 @@ def test_non_dict_record_is_skipped(self, tmp_path): ) assert set(StateStore(path).get_all_records()) == {"good:1"} + def test_record_without_primary_model_artifact_is_skipped(self, tmp_path): + path = tmp_path / "state.json" + record = make_sync_record("bad:1", DIGEST_A).to_json() + record["artifacts"][0]["role"] = "projector" + write_raw_state(path, {"schema_version": 2, "sync_records": {"bad:1": record}}) + + assert StateStore(path).get_all_records() == {} + def test_missing_sync_records_key(self, tmp_path): path = tmp_path / "state.json" write_raw_state(path, {"schema_version": 1}) @@ -94,7 +102,7 @@ def test_no_temp_file_left_behind(self, tmp_path): assert path.exists() assert not (tmp_path / "state.json.tmp").exists() payload = json.loads(path.read_text(encoding="utf-8")) - assert payload["schema_version"] == 1 + assert payload["schema_version"] == 2 assert "a:1" in payload["sync_records"] def test_overwrite_keeps_consistent_state(self, tmp_path): @@ -105,6 +113,39 @@ def test_overwrite_keeps_consistent_state(self, tmp_path): assert set(store.get_all_records()) == {"a:1", "b:1"} +class TestSchemaMigration: + def test_legacy_record_becomes_one_model_artifact(self, tmp_path): + path = tmp_path / "state.json" + legacy = { + "canonical_name": "m:1", + "digest": DIGEST_A, + "blob_path": "blob", + "import_alias_path": "alias.gguf", + "user_repo": "ollama/m", + "link_mode": "hard-link", + "imported_at": "2026-01-01T00:00:00+00:00", + "imported_model_path": "models/ollama/m/alias.gguf", + "import_command": ["lms", "import"], + } + write_raw_state(path, {"schema_version": 1, "sync_records": {"m:1": legacy}}) + + record = StateStore(path).get_record("m:1") + + assert record is not None + assert len(record.artifacts) == 1 + assert record.artifacts[0].role is ArtifactRole.MODEL + assert record.artifacts[0].digest == DIGEST_A + + def test_writes_schema_version_two(self, tmp_path): + path = tmp_path / "state.json" + StateStore(path).upsert(make_sync_record("m:1", DIGEST_A)) + + payload = json.loads(path.read_text(encoding="utf-8")) + + assert payload["schema_version"] == 2 + assert payload["sync_records"]["m:1"]["artifacts"][0]["role"] == "model" + + class TestRecordSchema: def test_record_serializes_link_mode(self, tmp_path): path = tmp_path / "state.json" diff --git a/tests/test_syncer.py b/tests/test_syncer.py index 7cf15f9..22d7cd8 100644 --- a/tests/test_syncer.py +++ b/tests/test_syncer.py @@ -9,7 +9,15 @@ write_manifest, ) -from studiolink.models import ImportMode, ImportResult, LinkMode, ModelReadiness +import studiolink.syncer as syncer_module +from studiolink.lmstudio_adapter import LMStudioError +from studiolink.models import ( + ArtifactRole, + ImportMode, + ImportResult, + LinkMode, + ModelReadiness, +) from studiolink.ollama_adapter import OllamaAdapter from studiolink.state import StateStore from studiolink.syncer import ( @@ -80,6 +88,260 @@ def test_direct_mode_imports_blob_path_without_alias(self, make_syncer): assert not config.import_staging_dir.exists() +class TestProjectorBundle: + def test_sync_imports_model_and_projector_into_one_repository(self, make_syncer): + syncer, config, fake = make_syncer() + projector_digest = "sha256:" + "c" * 64 + write_manifest( + config, + repository="llava", + content={ + "layers": [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": DIGEST_A, + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": projector_digest, + }, + ] + }, + ) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + model = scan_models(config)[0] + + result = syncer.sync([model], **sync_kwargs())[0] + + assert result.status == "synced" + assert len(fake.calls) == 2 + assert {call["user_repo"] for call in fake.calls} == {model.import_user_repo} + assert fake.calls[1]["source_path"].endswith( + model.artifact_import_filename(model.artifacts[1]) + ) + assert "/mmproj-" in fake.calls[1]["source_path"].replace("\\", "/") + record = StateStore(config.state_file).get_record(model.canonical_name) + assert record is not None + assert [artifact.role for artifact in record.artifacts] == [ + ArtifactRole.MODEL, + ArtifactRole.PROJECTOR, + ] + assert all( + artifact.imported_model_path is not None + and artifact.imported_model_path.is_file() + for artifact in record.artifacts + ) + + def test_waits_for_lm_studio_to_index_projector(self, make_syncer, monkeypatch): + syncer, config, fake = make_syncer() + projector_digest = "sha256:" + "c" * 64 + write_manifest( + config, + repository="llava", + content={ + "layers": [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": DIGEST_A, + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": projector_digest, + }, + ] + }, + ) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + model = scan_models(config)[0] + real_list = fake.list_models + calls = 0 + + def delayed_inventory(): + nonlocal calls + calls += 1 + return () if calls == 1 else real_list() + + monkeypatch.setattr(fake, "list_models", delayed_inventory) + monkeypatch.setattr(syncer_module.time, "sleep", lambda _: None) + + result = syncer.sync([model], **sync_kwargs())[0] + + assert result.status == "synced" + assert calls == 2 + + def test_resync_rechecks_projector_inventory(self, make_syncer, monkeypatch): + syncer, config, fake = make_syncer() + projector_digest = "sha256:" + "c" * 64 + write_manifest( + config, + repository="llava", + content={ + "layers": [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": DIGEST_A, + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": projector_digest, + }, + ] + }, + ) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + model = scan_models(config)[0] + assert syncer.sync([model], **sync_kwargs())[0].status == "synced" + fake.inventory_override = () + monkeypatch.setattr(syncer_module.time, "sleep", lambda _: None) + + result = syncer.sync([model], **sync_kwargs())[0] + + assert result.status == "error" + assert "vision-capable" in result.message + + def test_inventory_failure_is_reported_for_model(self, make_syncer, monkeypatch): + syncer, config, fake = make_syncer() + projector_digest = "sha256:" + "c" * 64 + write_manifest( + config, + repository="llava", + content={ + "layers": [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": DIGEST_A, + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": projector_digest, + }, + ] + }, + ) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + model = scan_models(config)[0] + + def fail_inventory(): + raise LMStudioError("inventory unavailable", ["lms"], 1, "boom") + + monkeypatch.setattr(fake, "list_models", fail_inventory) + + result = syncer.sync([model], **sync_kwargs())[0] + + assert result.status == "error" + assert "inventory unavailable" in result.message + record = StateStore(config.state_file).get_record(model.canonical_name) + assert record is not None + assert record.vision_confirmed is False + + def test_projector_failure_saves_progress_and_retry_resumes(self, make_syncer): + syncer, config, fake = make_syncer() + projector_digest = "sha256:" + "c" * 64 + write_manifest( + config, + repository="llava", + content={ + "layers": [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": DIGEST_A, + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": projector_digest, + }, + ] + }, + ) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + model = scan_models(config)[0] + projector_alias = config.import_staging_dir / model.artifact_import_filename( + model.artifacts[1] + ) + fake.fail_for = {str(projector_alias)} + + first = syncer.sync([model], **sync_kwargs())[0] + + assert first.status == "error" + partial = StateStore(config.state_file).get_record(model.canonical_name) + assert partial is not None + assert [artifact.role for artifact in partial.artifacts] == [ArtifactRole.MODEL] + + fake.fail_for.clear() + second = syncer.sync([model], **sync_kwargs())[0] + + assert second.status == "synced" + assert len(fake.calls) == 3 + assert fake.calls[-1]["source_path"] == str(projector_alias) + complete = StateStore(config.state_file).get_record(model.canonical_name) + assert complete is not None and len(complete.artifacts) == 2 + assert complete.vision_confirmed is True + + def test_missing_projector_prevents_every_import(self, make_syncer): + syncer, config, fake = make_syncer() + projector_digest = "sha256:" + "c" * 64 + write_manifest( + config, + repository="llava", + content={ + "layers": [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": DIGEST_A, + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": projector_digest, + }, + ] + }, + ) + write_blob(config, DIGEST_A) + model = scan_models(config)[0] + + result = syncer.sync([model], **sync_kwargs())[0] + + assert result.status == "error" + assert "ollama pull llava:1b" in result.message + assert fake.calls == [] + + def test_direct_mode_rejects_projector_bundle(self, make_syncer): + syncer, config, fake = make_syncer() + projector_digest = "sha256:" + "c" * 64 + write_manifest( + config, + repository="llava", + content={ + "layers": [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": DIGEST_A, + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": projector_digest, + }, + ] + }, + ) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + + result = syncer.sync( + scan_models(config), + **sync_kwargs(import_mode=ImportMode.DIRECT), + )[0] + + assert result.status == "error" + assert "--direct is not supported" in result.message + assert fake.calls == [] + + class TestCrossVolumeCopy: def test_copy_stages_on_ollama_volume_not_state_volume( self, make_syncer, monkeypatch From cc594de68ad8fd69723c3e9a89d0ef0fb5d004e8 Mon Sep 17 00:00:00 2001 From: Manan JK Date: Wed, 2 Sep 2026 02:11:48 +0530 Subject: [PATCH 3/7] fix: isolate alias failures per model and keep vision bundles co-located Alias creation: mkdir() and the stale-alias unlink() ran outside the OSError->RuntimeError conversion, so a PermissionError/FileExistsError escaped _sync_one and aborted the whole sync batch. Wrap both calls so each model reports an error result instead. Artifact order: a manifest that lists the projector before the model made per-artifact partial persistence serialize a projector-only record, whose primary_artifact lookup raised StopIteration and aborted the batch. The adapter now normalizes model layers first, partial persistence never serializes without a model artifact, and primary_artifact raises a clear ValueError if the invariant is ever violated. Bundle co-location: re-pulling one artifact of a vision bundle changed import_user_repo, but the unchanged artifact was still reused from the old digest-keyed directory, splitting the pair across LM Studio directories. Reuse now requires the prior target to live in the current destination directory, so both artifacts are re-imported together and vision confirmation checks the new location. Already-synced policy: a complete import that Ollama can no longer back now reports "skipped" for bundles too, matching text models, instead of a spurious stale error. same_file(): treat st_ino == 0 as unknowable identity. On filesystems without file ids (FAT/exFAT) every file on the volume compared equal, which could silently reuse a stale alias and import old weights. --- src/studiolink/models.py | 9 +- src/studiolink/ollama_adapter.py | 4 + src/studiolink/syncer.py | 98 +++++++++----- tests/test_syncer.py | 214 +++++++++++++++++++++++++++++++ 4 files changed, 290 insertions(+), 35 deletions(-) diff --git a/src/studiolink/models.py b/src/studiolink/models.py index 73f065e..8eccad4 100644 --- a/src/studiolink/models.py +++ b/src/studiolink/models.py @@ -210,10 +210,11 @@ class SyncRecord: @property def primary_artifact(self) -> SyncedArtifact: - return next( - artifact - for artifact in self.artifacts - if artifact.role is ArtifactRole.MODEL + for artifact in self.artifacts: + if artifact.role is ArtifactRole.MODEL: + return artifact + raise ValueError( + f"sync record for {self.canonical_name!r} has no model artifact" ) # Compatibility properties for callers and schema-v1 JSON consumers. diff --git a/src/studiolink/ollama_adapter.py b/src/studiolink/ollama_adapter.py index f5783f8..9ff0250 100644 --- a/src/studiolink/ollama_adapter.py +++ b/src/studiolink/ollama_adapter.py @@ -156,6 +156,10 @@ def _parse_manifest(self, manifest_path: Path) -> OllamaModel | None: for role in sorted(unsupported_roles): issues.append(f"Ollama {role} layers are not supported") + # Import order is semantic: partial sync records must always contain + # the model artifact, so normalize model layers before projectors + # regardless of the order the manifest lists them in. + artifact_layers.sort(key=lambda entry: entry[2] is not ArtifactRole.MODEL) artifacts = tuple( self._parse_artifact(layer, role, index) for index, layer, role in artifact_layers diff --git a/src/studiolink/syncer.py b/src/studiolink/syncer.py index 03525df..6a7b4d8 100644 --- a/src/studiolink/syncer.py +++ b/src/studiolink/syncer.py @@ -37,6 +37,11 @@ def same_file(left: Path, right: Path) -> bool: left_stat, right_stat = left.stat(), right.stat() except OSError: return False + if left_stat.st_ino == 0 or right_stat.st_ino == 0: + # Filesystems without file ids (FAT/exFAT) report inode 0, which + # would make every file on the volume compare equal and reuse a + # stale alias. Identity is unknowable there, so never match. + return False return (left_stat.st_dev, left_stat.st_ino) == ( right_stat.st_dev, right_stat.st_ino, @@ -113,24 +118,30 @@ def _sync_one( message="already synced according to StudioLink state", record=existing, ) - if model.readiness is ModelReadiness.STALE: - logger.debug("Model %s has a missing artifact", model.canonical_name) - return SyncResult( - model=model, - status="error", - message=( - "model artifact is missing from the Ollama blob store; " - f"run `ollama pull {model.canonical_name}` to restore it" - ), - ) - if model.readiness is ModelReadiness.INVALID: - logger.debug("Model %s is invalid: %s", model.canonical_name, model.issues) - return SyncResult( - model=model, - status="error", - message="model is not ready for import: " - + "; ".join(model.issues or ("unknown error",)), - ) + if not already_synced: + # A complete import that Ollama can no longer back is still fine + # to keep; only a model we are about to (re-)import must be + # blocked by stale or invalid artifacts. + if model.readiness is ModelReadiness.STALE: + logger.debug("Model %s has a missing artifact", model.canonical_name) + return SyncResult( + model=model, + status="error", + message=( + "model artifact is missing from the Ollama blob store; " + f"run `ollama pull {model.canonical_name}` to restore it" + ), + ) + if model.readiness is ModelReadiness.INVALID: + logger.debug( + "Model %s is invalid: %s", model.canonical_name, model.issues + ) + return SyncResult( + model=model, + status="error", + message="model is not ready for import: " + + "; ".join(model.issues or ("unknown error",)), + ) artifacts = self._model_artifacts(model) if import_mode is ImportMode.DIRECT and len(artifacts) > 1: @@ -147,6 +158,7 @@ def _sync_one( (artifact.role, artifact.digest): artifact for artifact in (existing.artifacts if existing is not None else ()) } + destination_dir = self.config.lmstudio_models_dir / model.import_user_repo completed: list[SyncedArtifact] = [] last_result: ImportResult | None = None imported_at = ( @@ -156,13 +168,14 @@ def _sync_one( ) for artifact in artifacts: - if artifact.digest is None or artifact.blob_path is None: - return SyncResult( - model=model, - status="error", - message=f"{artifact.role.value} artifact is not importable", - ) - prior = existing_by_identity.get((artifact.role, artifact.digest)) + # A None digest can never match a stored record (records always + # carry a real digest), so the importability check below reports + # it after the reuse lookup. + prior = ( + existing_by_identity.get((artifact.role, artifact.digest)) + if artifact.digest is not None + else None + ) prior_target = prior.imported_model_path if prior is not None else None if prior is not None and prior_target is None: prior_target = ( @@ -174,6 +187,11 @@ def _sync_one( prior is not None and prior_target is not None and prior_target.is_file() + # A changed sibling digest also changes import_user_repo. + # Reusing the artifact from the old bundle directory would + # split the pair across LM Studio directories and break + # vision pairing, so re-import into the current destination. + and prior_target.parent == destination_dir ): completed.append( SyncedArtifact( @@ -187,6 +205,13 @@ def _sync_one( ) continue + if artifact.digest is None or artifact.blob_path is None: + return SyncResult( + model=model, + status="error", + message=f"{artifact.role.value} artifact is not importable", + ) + if import_mode is ImportMode.DIRECT: alias_path = artifact.blob_path alias_created = False @@ -257,7 +282,11 @@ def _sync_one( import_command=import_result.command, ) ) - if not dry_run: + if not dry_run and any( + artifact.role is ArtifactRole.MODEL for artifact in completed + ): + # Never persist a partial without the model artifact: its + # serialization requires a primary. partial = self._make_record( model, completed, link_mode, imported_at, vision_confirmed=False ) @@ -358,7 +387,9 @@ def _partial_result( dry_run: bool, ) -> SyncResult: record = None - if completed: + if completed and any( + artifact.role is ArtifactRole.MODEL for artifact in completed + ): record = self._make_record( model, completed, @@ -454,14 +485,19 @@ def _ensure_import_alias( # The alias survived a re-pull but now points at the old blob content; # replace it so we never import stale weights. logger.debug("Replacing stale import alias: %s", alias_path) - alias_path.unlink() - staging_dir.mkdir(parents=True, exist_ok=True) + try: + alias_path.unlink() + except OSError as exc: + raise RuntimeError( + f"Failed to replace stale import alias {alias_path}: {exc}" + ) from exc try: + staging_dir.mkdir(parents=True, exist_ok=True) os.link(blob_path, alias_path) - logger.debug("Created hard link: %s -> %s", alias_path, blob_path) - return alias_path, True except OSError as exc: raise RuntimeError(f"Failed to create import alias (hard link): {exc}") from exc + logger.debug("Created hard link: %s -> %s", alias_path, blob_path) + return alias_path, True def _format_import_message(result: ImportResult) -> str: diff --git a/tests/test_syncer.py b/tests/test_syncer.py index 22d7cd8..5398e6b 100644 --- a/tests/test_syncer.py +++ b/tests/test_syncer.py @@ -1,9 +1,12 @@ import errno import os +import types +from pathlib import Path from conftest import ( DIGEST_A, DIGEST_B, + blob_filename, make_ollama_model, write_blob, write_manifest, @@ -588,3 +591,214 @@ def test_sync_result_carries_model(self, make_syncer): results = syncer.sync([model], **sync_kwargs()) assert results[0].model is model assert results[0].model.readiness is ModelReadiness.READY + + +class TestAliasFilesystemErrors: + def test_blocked_staging_dir_is_per_model_error(self, make_syncer): + # Regression: PermissionError/FileExistsError from mkdir() or the + # stale-alias unlink() escaped _sync_one and aborted the whole batch. + syncer, config, fake = make_syncer() + write_manifest(config, repository="m1", tag="1b", digest=DIGEST_A) + write_manifest(config, repository="m2", tag="1b", digest=DIGEST_B) + write_blob(config, DIGEST_A) + write_blob(config, DIGEST_B) + config.import_staging_dir.parent.mkdir(parents=True, exist_ok=True) + config.import_staging_dir.write_text("a file, not a directory") + + results = syncer.sync(scan_models(config), **sync_kwargs()) + + assert [result.status for result in results] == ["error", "error"] + assert all("import alias" in result.message for result in results) + assert fake.calls == [] + + def test_stale_alias_unlink_failure_is_per_model_error( + self, make_syncer, monkeypatch + ): + syncer, config, fake = make_syncer() + write_manifest(config, digest=DIGEST_A) + write_blob(config, DIGEST_A) + model = scan_models(config)[0] + config.import_staging_dir.mkdir(parents=True) + stale = config.import_staging_dir / model.import_filename + stale.write_bytes(b"stale content") + + def denied(path, *args, **kwargs): + if path == stale: + raise PermissionError(1, "Operation not permitted") + return real_unlink(path, *args, **kwargs) + + real_unlink = Path.unlink + monkeypatch.setattr(Path, "unlink", denied) + + result = syncer.sync([model], **sync_kwargs())[0] + + assert result.status == "error" + assert "stale import alias" in result.message + + def test_zero_inode_files_never_compare_equal(self, tmp_path, monkeypatch): + # Filesystems without file ids (FAT/exFAT) report st_ino == 0 for + # every file; identity must be treated as unknowable rather than + # matching every file on the volume (stale-weight reuse risk). + left = tmp_path / "a.gguf" + right = tmp_path / "b.gguf" + left.write_bytes(b"GGUFsame-size") + right.write_bytes(b"GGUFsame-size") + real_stat = os.stat + + def inodeless_stat(path, *args, **kwargs): + st = real_stat(path, *args, **kwargs) + return types.SimpleNamespace( + st_dev=st.st_dev, + st_ino=0, + st_size=st.st_size, + st_mtime_ns=st.st_mtime_ns, + ) + + monkeypatch.setattr(Path, "stat", inodeless_stat) + + assert same_file(left, right) is False + assert same_file(left, left) is False + + +def write_bundle(config, model_digest, projector_digest, *, model_first): + layers = [ + {"mediaType": "application/vnd.ollama.image.model", "digest": model_digest}, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": projector_digest, + }, + ] + if not model_first: + layers.reverse() + write_manifest(config, repository="llava", content={"layers": layers}) + + +class TestProjectorFirstManifests: + def test_projector_first_manifest_imports_model_first(self, make_syncer): + # Regression: a partial record persisted before the model artifact + # made primary_artifact raise StopIteration and abort the batch. + syncer, config, fake = make_syncer() + projector_digest = "sha256:" + "c" * 64 + write_bundle(config, DIGEST_A, projector_digest, model_first=False) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + model = scan_models(config)[0] + + result = syncer.sync([model], **sync_kwargs())[0] + + assert result.status == "synced" + assert [artifact["user_repo"] for artifact in fake.calls] == [ + model.import_user_repo + ] * 2 + assert fake.calls[0]["source_path"].endswith(model.import_filename) + record = StateStore(config.state_file).get_record(model.canonical_name) + assert record is not None + assert [artifact.role for artifact in record.artifacts] == [ + ArtifactRole.MODEL, + ArtifactRole.PROJECTOR, + ] + + def test_failed_model_import_in_projector_first_batch_continues(self, make_syncer): + syncer, config, fake = make_syncer() + projector_digest = "sha256:" + "c" * 64 + write_bundle(config, DIGEST_A, projector_digest, model_first=False) + write_manifest(config, repository="mistral", tag="7b", digest=DIGEST_B) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + write_blob(config, DIGEST_B) + models = {m.canonical_name: m for m in scan_models(config)} + fake.fail_for = { + str(config.import_staging_dir / models["llava:1b"].import_filename) + } + + results = syncer.sync( + [models["llava:1b"], models["mistral:7b"]], **sync_kwargs() + ) + + assert results[0].status == "error" + assert results[0].record is None # no projector-only partial persisted + assert results[1].status == "synced" + records = StateStore(config.state_file).get_all_records() + assert set(records) == {"mistral:7b"} + + +class TestBundleCoLocation: + """A changed sibling digest changes import_user_repo; both artifacts must + end up in the new bundle directory or LM Studio cannot pair them.""" + + def test_changed_model_digest_reimports_whole_bundle(self, make_syncer): + syncer, config, fake = make_syncer() + projector_digest = "sha256:" + "c" * 64 + write_bundle(config, DIGEST_A, projector_digest, model_first=True) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + model = scan_models(config)[0] + assert syncer.sync([model], **sync_kwargs())[0].status == "synced" + + write_blob(config, DIGEST_B) + write_bundle(config, DIGEST_B, projector_digest, model_first=True) + model = scan_models(config)[0] + + result = syncer.sync([model], **sync_kwargs())[0] + + assert result.status == "synced" + new_repo = model.import_user_repo + assert [call["user_repo"] for call in fake.calls[2:]] == [new_repo, new_repo] + record = StateStore(config.state_file).get_record("llava:1b") + assert record is not None + assert all( + artifact.imported_model_path is not None + and artifact.imported_model_path.parent + == config.lmstudio_models_dir / new_repo + for artifact in record.artifacts + ) + + def test_changed_projector_digest_reimports_whole_bundle(self, make_syncer): + # The dangerous direction: the unchanged model must move to the new + # bundle directory too, and vision must be confirmed against it. + syncer, config, fake = make_syncer() + projector_digest = "sha256:" + "c" * 64 + write_bundle(config, DIGEST_A, projector_digest, model_first=True) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + model = scan_models(config)[0] + assert syncer.sync([model], **sync_kwargs())[0].status == "synced" + + new_projector = "sha256:" + "d" * 64 + write_blob(config, new_projector) + write_bundle(config, DIGEST_A, new_projector, model_first=True) + model = scan_models(config)[0] + + result = syncer.sync([model], **sync_kwargs())[0] + + assert result.status == "synced" + new_repo = model.import_user_repo + assert [call["user_repo"] for call in fake.calls[2:]] == [new_repo, new_repo] + assert fake.calls[3]["source_path"].endswith( + model.artifact_import_filename(model.artifacts[1]) + ) + record = StateStore(config.state_file).get_record("llava:1b") + assert record is not None + assert all( + artifact.imported_model_path is not None + and artifact.imported_model_path.parent + == config.lmstudio_models_dir / new_repo + for artifact in record.artifacts + ) + + def test_synced_bundle_skips_when_blob_is_removed(self, make_syncer): + # An intact LM Studio import needs nothing from Ollama; "already + # synced" must win over the stale check for bundles too. + syncer, config, fake = make_syncer() + projector_digest = "sha256:" + "c" * 64 + write_bundle(config, DIGEST_A, projector_digest, model_first=True) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + model = scan_models(config)[0] + assert syncer.sync([model], **sync_kwargs())[0].status == "synced" + (config.ollama_blobs_dir / blob_filename(projector_digest)).unlink() + + result = syncer.sync([model], **sync_kwargs())[0] + + assert result.status == "skipped" + assert len(fake.calls) == 2 From fec96831b51e59a18da46d5f40f329ae88bf6e59 Mon Sep 17 00:00:00 2001 From: Manan JK Date: Wed, 2 Sep 2026 02:12:01 +0530 Subject: [PATCH 4/7] fix: make state persistence durable and BOM-tolerant Read state.json with utf-8-sig so an editor-added BOM no longer makes every record disappear and trigger a mass re-import. Write through a per-process temp file with fsync before the atomic replace, so concurrent writers cannot interleave into one temp file and a crash cannot leave a truncated state file behind. --- src/studiolink/state.py | 22 +++++++++++++++++----- tests/test_state.py | 28 ++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 5 deletions(-) diff --git a/src/studiolink/state.py b/src/studiolink/state.py index 3e661f4..a928504 100644 --- a/src/studiolink/state.py +++ b/src/studiolink/state.py @@ -61,19 +61,31 @@ def _parse_record(name: str, record_data: object) -> SyncRecord | None: return None def _save_raw(self, records: Mapping[str, object]) -> None: - """Persist raw JSON dict to disk atomically.""" + """Persist raw JSON dict to disk atomically and durably.""" self.path.parent.mkdir(parents=True, exist_ok=True) payload = json.dumps({"schema_version": 2, "sync_records": records}, indent=2) - tmp_path = self.path.parent / (self.path.name + ".tmp") - tmp_path.write_text(payload, encoding="utf-8") - os.replace(tmp_path, self.path) + # A per-process temp name keeps concurrent writers from interleaving + # writes into one file, and fsync before the replace keeps a crash + # from leaving a truncated state file behind. + tmp_path = self.path.parent / f"{self.path.name}.tmp-{os.getpid()}" + try: + with open(tmp_path, "w", encoding="utf-8") as handle: + handle.write(payload) + handle.flush() + os.fsync(handle.fileno()) + os.replace(tmp_path, self.path) + except BaseException: + tmp_path.unlink(missing_ok=True) + raise def _load_raw(self) -> dict[str, object]: """Load raw JSON dict from disk.""" if not self.path.exists(): return {} try: - payload = json.loads(self.path.read_text(encoding="utf-8")) + # utf-8-sig tolerates editors that save the file with a BOM; + # a strict read would discard every record as "corrupted". + payload = json.loads(self.path.read_text(encoding="utf-8-sig")) except json.JSONDecodeError as exc: logger.warning( "Corrupted state file at %s: %s. Starting fresh.", self.path, exc diff --git a/tests/test_state.py b/tests/test_state.py index 826d7db..bb27e58 100644 --- a/tests/test_state.py +++ b/tests/test_state.py @@ -157,3 +157,31 @@ def test_record_serializes_link_mode(self, tmp_path): def test_record_from_json_requires_fields(self): with pytest.raises(KeyError): SyncRecord.from_json({"canonical_name": "m:1"}) + + +class TestStateDurability: + def test_loads_state_saved_with_utf8_bom(self, tmp_path): + # Regression: an editor that saves state.json with a BOM used to make + # every record disappear ("Starting fresh") and re-import everything. + path = tmp_path / "state.json" + record = make_sync_record("llama3:1b", DIGEST_A) + StateStore(path).upsert(record) + raw = path.read_bytes() + assert not raw.startswith(b"\xef\xbb\xbf") + path.write_bytes(b"\xef\xbb\xbf" + raw) + + restored = StateStore(path).get_record("llama3:1b") + + assert restored is not None + assert restored.canonical_name == "llama3:1b" + assert restored.digest == DIGEST_A + + def test_save_leaves_no_temp_files_behind(self, tmp_path): + path = tmp_path / "state.json" + store = StateStore(path) + + store.upsert(make_sync_record("llama3:1b", DIGEST_A)) + store.upsert(make_sync_record("llama3:2b", DIGEST_A)) + + assert [item.name for item in path.parent.iterdir()] == [path.name] + assert len(store.get_all_records()) == 2 From 2efb8b3564d592c3ea9a167ca3a45217e7b0d73a Mon Sep 17 00:00:00 2001 From: Manan JK Date: Wed, 2 Sep 2026 02:12:01 +0530 Subject: [PATCH 5/7] fix: degrade status gracefully and probe doctor prerequisites status() no longer crashes when lms cannot be executed: with a broken CLI, projector bundles read as pending instead of the whole command failing for users with synced vision models. doctor gains two checks: a reversible hard-link probe of a real blob into each staging destination (catching restrictions such as Linux fs.protected_hardlinks that same-volume st_dev checks cannot see), and an lms ls --json inventory check that vision verification depends on. prune now also sweeps .studiolink-imports, where copy-mode temporary aliases live; anything found there survived a crash or cleanup failure and pinned blob space. --- README.md | 2 + src/studiolink/service.py | 100 ++++++++++++++++++++++++- tests/test_service.py | 152 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 250 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index cef8326..51cd231 100644 --- a/README.md +++ b/README.md @@ -164,6 +164,8 @@ sdl doctor --json - Ollama manifests and blobs directories exist - LM Studio models directory exists - Hard-link filesystem/volume compatibility +- Hard-link permission on blobs (a real link is created and removed again, catching restrictions such as Linux `fs.protected_hardlinks`) +- LM Studio model inventory (`lms ls --json`; required to verify projector bundles) - Discovered Ollama models - Blob presence validation - GGUF header validation diff --git a/src/studiolink/service.py b/src/studiolink/service.py index b736c48..1fab354 100644 --- a/src/studiolink/service.py +++ b/src/studiolink/service.py @@ -7,7 +7,7 @@ from typing import TYPE_CHECKING from studiolink.config import StudioLinkConfig -from studiolink.lmstudio_adapter import LMStudioAdapter +from studiolink.lmstudio_adapter import LMStudioAdapter, LMStudioError from studiolink.models import ( ArtifactRole, DoctorCheck, @@ -85,7 +85,18 @@ def status(self) -> list[StatusEntry]: and records.get(model.canonical_name) is not None for model in models ) - inventory = self.lmstudio.list_models() if needs_inventory else None + inventory: tuple[LMStudioModel, ...] | None = None + if needs_inventory: + try: + inventory = self.lmstudio.list_models() + except LMStudioError as exc: + # A broken lms must not take the whole status command down; + # without inventory evidence vision bundles read as pending. + logger.warning( + "Could not list LM Studio models (%s); projector bundles " + "will be reported as pending", + exc, + ) return [ StatusEntry( model=model, @@ -174,8 +185,20 @@ def prune(self, *, dry_run: bool = False) -> PruneReport: } alias_results: list[PruneResult] = [] - staging = self.config.import_staging_dir - if staging.exists(): + # Copy mode stages temporary aliases inside the Ollama store; they are + # removed after each import, so anything found there survived a crash + # or a cleanup failure and pins blob space until it is swept. + staging_dirs = list( + dict.fromkeys( + [ + self.config.import_staging_dir, + self.config.ollama_models_dir / ".studiolink-imports", + ] + ) + ) + for staging in staging_dirs: + if not staging.exists(): + continue for path in sorted(staging.iterdir()): if not path.is_file(): continue @@ -267,6 +290,7 @@ def doctor(self) -> list[DoctorCheck]: str(self.config.lmstudio_models_dir), ), self._volume_compatibility_check(), + self._hard_link_permission_check(), ] try: @@ -297,6 +321,19 @@ def doctor(self) -> list[DoctorCheck]: logger.debug("Doctor: LM Studio capabilities check failed: %s", exc) checks.append(DoctorCheck("lm studio import capabilities", False, str(exc))) + try: + inventory = self.lmstudio.list_models() + checks.append( + DoctorCheck( + "lm studio model inventory", + True, + f"{len(inventory)} model(s) indexed", + ) + ) + except Exception as exc: + logger.debug("Doctor: LM Studio inventory check failed: %s", exc) + checks.append(DoctorCheck("lm studio model inventory", False, str(exc))) + models = self.scan() checks.append( DoctorCheck( @@ -373,6 +410,61 @@ def _volume_compatibility_check(self) -> DoctorCheck: ) return DoctorCheck("hard-link volume compatibility", ok, details) + def _hard_link_permission_check(self) -> DoctorCheck: + """Probe that a real blob can actually be hard-linked into staging. + + Same-volume checks (st_dev) cannot catch policies such as Linux + fs.protected_hardlinks, which blocks linking a blob the user cannot + write - the typical layout when Ollama runs as a system service. The + probe links a real blob under a temporary name and removes it again. + """ + name = "hard-link permission on blobs" + blob = self._any_blob_file() + if blob is None: + return DoctorCheck(name, True, "skipped: no blobs present") + destinations = list( + dict.fromkeys( + [ + self.config.import_staging_dir, + self.config.ollama_models_dir / ".studiolink-imports", + ] + ) + ) + outcomes: list[str] = [] + ok = True + for destination in destinations: + if not destination.is_dir(): + outcomes.append(f"{destination}: skipped (not created yet)") + continue + probe = destination / f".studiolink-probe-{os.getpid()}" + try: + probe.unlink(missing_ok=True) + os.link(blob, probe) + except OSError as exc: + ok = False + outcomes.append( + f"{destination}: FAILED ({exc}); blobs must be linkable " + "here - on Linux check blob ownership and " + "fs.protected_hardlinks, or use --copy" + ) + else: + outcomes.append(f"{destination}: ok") + finally: + try: + probe.unlink(missing_ok=True) + except OSError: + pass + return DoctorCheck(name, ok, "; ".join(outcomes)) + + def _any_blob_file(self) -> Path | None: + blobs = self.config.ollama_blobs_dir + if not blobs.is_dir(): + return None + for path in sorted(blobs.iterdir()): + if path.is_file(): + return path + return None + def _record_is_synced( self, model: OllamaModel, diff --git a/tests/test_service.py b/tests/test_service.py index c4b02ef..b058ae0 100644 --- a/tests/test_service.py +++ b/tests/test_service.py @@ -12,6 +12,7 @@ ) import studiolink.service as service_module +from studiolink.lmstudio_adapter import LMStudioError from studiolink.models import LinkMode from studiolink.ollama_adapter import OllamaAdapter from studiolink.service import StudioLinkService @@ -501,3 +502,154 @@ def test_select_models_matches_repository_all_tags( selected = service._select_models(models, ["llama3"]) assert {m.tag for m in selected} == {"1b", "70b"} + + +class TestStatusInventoryDegradation: + def make_synced_vision_model(self, config, service): + projector_digest = "sha256:" + "c" * 64 + write_manifest( + config, + repository="llava", + content={ + "layers": [ + { + "mediaType": "application/vnd.ollama.image.model", + "digest": DIGEST_A, + }, + { + "mediaType": "application/vnd.ollama.image.projector", + "digest": projector_digest, + }, + ] + }, + ) + write_blob(config, DIGEST_A) + write_blob(config, projector_digest) + model = OllamaAdapter(config).scan_models()[0] + result = service.sync(model_names=[model.canonical_name])[0] + assert result.status == "synced" + return model + + def test_status_survives_lm_studio_inventory_failure( + self, make_config, fake_lmstudio, monkeypatch + ): + # Regression: a broken lms used to crash the whole status command for + # users with synced vision models. + config = make_config() + fake_lmstudio.models_dir = config.lmstudio_models_dir + service = build_service(config, fake_lmstudio) + self.make_synced_vision_model(config, service) + + def broken_inventory(): + raise LMStudioError("lms is broken", ["lms"], 1, "boom") + + monkeypatch.setattr(fake_lmstudio, "list_models", broken_inventory) + + entries = service.status() + + entry = next(e for e in entries if e.model.canonical_name == "llava:1b") + assert entry.synced is False + assert entry.display_status == "pending" + + def test_status_uses_inventory_when_available(self, make_config, fake_lmstudio): + config = make_config() + fake_lmstudio.models_dir = config.lmstudio_models_dir + service = build_service(config, fake_lmstudio) + self.make_synced_vision_model(config, service) + + entries = service.status() + + entry = next(e for e in entries if e.model.canonical_name == "llava:1b") + assert entry.synced is True + + +class TestDoctorHardLinkPermission: + def test_probe_passes_on_linkable_filesystem(self, make_config, fake_lmstudio): + config = make_config() + write_manifest(config, digest=DIGEST_A) + write_blob(config, DIGEST_A) + config.import_staging_dir.mkdir(parents=True) + service = build_service(config, fake_lmstudio) + + checks = service.doctor() + + check = next(c for c in checks if c.name == "hard-link permission on blobs") + assert check.ok is True + assert "ok" in check.details + assert not any( + probe.name.startswith(".studiolink-probe-") + for probe in config.import_staging_dir.iterdir() + ) + + def test_probe_reports_unlinkable_blobs( + self, make_config, fake_lmstudio, monkeypatch + ): + config = make_config() + write_manifest(config, digest=DIGEST_A) + write_blob(config, DIGEST_A) + config.import_staging_dir.mkdir(parents=True) + service = build_service(config, fake_lmstudio) + + def denied(source, target, *args, **kwargs): + raise PermissionError(1, "Operation not permitted") + + monkeypatch.setattr(service_module.os, "link", denied) + + checks = service.doctor() + + check = next(c for c in checks if c.name == "hard-link permission on blobs") + assert check.ok is False + assert "FAILED" in check.details + assert "fs.protected_hardlinks" in check.details + + def test_probe_skips_without_blobs(self, make_config, fake_lmstudio): + config = make_config() + for directory in ( + config.ollama_manifests_dir, + config.ollama_blobs_dir, + config.lmstudio_models_dir, + ): + directory.mkdir(parents=True) + service = build_service(config, fake_lmstudio) + + checks = service.doctor() + + check = next(c for c in checks if c.name == "hard-link permission on blobs") + assert check.ok is True + assert "skipped" in check.details + + def test_doctor_checks_lm_studio_inventory(self, make_config, fake_lmstudio): + config = make_config() + config.ollama_manifests_dir.mkdir(parents=True) + config.ollama_blobs_dir.mkdir(parents=True) + config.lmstudio_models_dir.mkdir(parents=True) + service = build_service(config, fake_lmstudio) + + checks = service.doctor() + + check = next(c for c in checks if c.name == "lm studio model inventory") + assert check.ok is True + assert "indexed" in check.details + + +class TestPruneSweepsCopyStaging: + def test_sweeps_copy_staging_leftovers(self, make_config, fake_lmstudio): + # Copy-mode aliases are removed after each import; anything found in + # .studiolink-imports survived a crash and pins blob space. + config = make_config() + service = build_service(config, fake_lmstudio) + write_manifest(config, digest=DIGEST_A) + blob = write_blob(config, DIGEST_A) + model = OllamaAdapter(config).scan_models()[0] + copy_staging = config.ollama_models_dir / ".studiolink-imports" + copy_staging.mkdir(parents=True) + leftover = copy_staging / "crashed-leftover.gguf" + leftover.write_bytes(b"GGUFxxxx") + healthy = copy_staging / model.import_filename + os.link(blob, healthy) + + report = service.prune() + + removed = {result.path for result in report.aliases if result.removed} + assert leftover in removed + assert healthy.exists() From 4dccc178154740a84515dc085edb4483241f97d1 Mon Sep 17 00:00:00 2001 From: Manan JK Date: Wed, 2 Sep 2026 02:12:02 +0530 Subject: [PATCH 6/7] fix: compare release versions by leading digits and pin pip upgrade _version_tuple concatenated every digit run in a component, so 0.1.0rc1 compared as (0, 1, 11) - newer than any 0.1.x release. Only the leading numeric run counts now, so pre-releases never look newer than their release. The upgrade command pins the exact version it just reported, so the success message cannot describe a different release than the one pip installed. --- src/studiolink/cli.py | 11 ++++++++--- tests/test_cli.py | 7 ++++++- 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/src/studiolink/cli.py b/src/studiolink/cli.py index 6efc2ab..25c800e 100644 --- a/src/studiolink/cli.py +++ b/src/studiolink/cli.py @@ -3,6 +3,7 @@ import argparse import json import logging +import re import subprocess import sys import urllib.request @@ -297,7 +298,9 @@ def run_upgrade(args: argparse.Namespace, service: StudioLinkService) -> int: "-m", "pip", "install", - "studiolink", + # Pin the version we just reported so the success message cannot + # describe a different release than the one pip actually installed. + f"studiolink=={latest}", "--upgrade", "--quiet", ] @@ -329,8 +332,10 @@ def run_upgrade(args: argparse.Namespace, service: StudioLinkService) -> int: def _version_tuple(version: str) -> tuple[int, ...]: parts: list[int] = [] for piece in version.split("."): - digits = "".join(ch for ch in piece if ch.isdigit()) - parts.append(int(digits) if digits else 0) + # Compare only the leading numeric run of each component so that + # pre-release suffixes like "1rc1" do not concatenate into 11. + match = re.match(r"\d+", piece) + parts.append(int(match.group()) if match else 0) return tuple(parts) diff --git a/tests/test_cli.py b/tests/test_cli.py index 02379a9..48907c2 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -249,7 +249,12 @@ def test_multi_digit_patch_compares_numerically(self): assert cli._version_tuple("1.0.0") > cli._version_tuple("0.99.99") def test_versions_with_suffixes_do_not_crash(self): - assert cli._version_tuple("0.1.0rc1") == (0, 1, 1) + # Regression: "1rc1" used to concatenate its digits into 11, so a + # pre-release compared as newer than any 0.1.x release. Suffixes are + # now truncated, so a pre-release never looks newer than its release. + assert cli._version_tuple("0.1.0rc1") == (0, 1, 0) + assert not cli._version_tuple("0.1.0rc1") > cli._version_tuple("0.1.0") + assert cli._version_tuple("0.1.1") > cli._version_tuple("0.1.0rc1") class TestUpgradeCommand: From 191b462c053b036589f71d25f419295bef799ca4 Mon Sep 17 00:00:00 2001 From: Manan JK Date: Wed, 2 Sep 2026 02:12:02 +0530 Subject: [PATCH 7/7] docs: add reliability audit findings; release 0.3.1 The audit documents the verified failure modes behind this branch's fixes, including two merge blockers in the bundles branch (projector-first manifests and split vision bundles) found during review. --- docs/research/branch-reliability-audit.md | 198 ++++++++++++++++++++++ src/studiolink/__init__.py | 2 +- 2 files changed, 199 insertions(+), 1 deletion(-) create mode 100644 docs/research/branch-reliability-audit.md diff --git a/docs/research/branch-reliability-audit.md b/docs/research/branch-reliability-audit.md new file mode 100644 index 0000000..8174bb7 --- /dev/null +++ b/docs/research/branch-reliability-audit.md @@ -0,0 +1,198 @@ +# Branch reliability audit + +**Date:** 2026-09-02 +**Scope:** `main` (`221229d`), `stabilize-sync-lifecycle` (`3d5c60b`), and +`support-artifact-bundles` (`1090dbc`) +**Method:** read-only `git show`/`git diff` inspection, the existing worktrees, +repository tests, small temporary-directory probes, and primary platform sources. + +## Executive conclusion + +The proposed four fixes are well chosen, but the bundles branch has two additional +merge blockers: + +- Re-pulling a vision bundle when only one artifact digest changes reuses the + unchanged artifact from the old digest-keyed LM Studio directory and imports + the changed artifact into the new directory. The split can either fail forever + or, worse, be falsely confirmed against the old bundle. +- A valid manifest that lists the projector before the model imports the + projector first, then tries to persist a projector-only partial record. + `primary_artifact` raises `StopIteration`, aborting the entire batch. + +Recommended pre-merge order: + +1. **P1:** normalize bundle imports to model-first and never serialize a partial + `SyncRecord` without exactly one model artifact. +2. **P1:** keep every artifact of a changed vision bundle co-located; do not reuse + an unchanged artifact from a different `import_user_repo` without importing or + linking it into the new repository. +3. **P1:** translate stale-alias `unlink()` and staging `mkdir()` failures to the + per-model `RuntimeError` result, with two-model continuation tests. +4. **P2:** make bundles `status()` degrade to pending when `lms ls --json` fails. +5. **P2:** read state with `utf-8-sig`, with a real-record regression test. +6. **P2/platform:** add an actual hard-link permission probe and actionable Linux + error; `st_dev` equality alone is insufficient. +7. **P2/conditional correctness:** make `same_file()` return false when either + inode/file identifier is zero. A size comparison is only an extra rejection + check, not proof that equal-sized files are the same file. + +The full existing suites pass (main 117, stabilize 135, bundles 156), so the +listed failures are coverage gaps rather than known failing tests. + +## Branch relationship + +`support-artifact-bundles` is not an alternative to `stabilize-sync-lifecycle`: +its merge base with stabilize is exactly `3d5c60b`, so bundles contains all of +stabilize plus `7bbf601` and `1090dbc`. Review and merge planning should treat +the work as one stack. + +## Verified bugs + +| Claim | Verdict and evidence | +|---|---| +| Batch abort on alias filesystem errors | **Confirmed on all three refs.** The stale `unlink()` and `mkdir()` occur before the `try` that wraps only `os.link()`: [main 198-204](https://github.com/ensevengg/StudioLink/blob/221229ddd9df6176134a1da695fd82c0a21499cb/src/studiolink/syncer.py#L198-L204), [stabilize 231-237](https://github.com/ensevengg/StudioLink/blob/3d5c60b1bc6b3927b1fb2be196d0fd87c591526c/src/studiolink/syncer.py#L231-L237), [bundles 457-463](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L457-L463). Callers catch only `RuntimeError` ([bundles 194-207](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L194-L207)), while the outer batch loop has no catch ([74-80](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L74-L80)). A raw `PermissionError`/`FileExistsError` therefore aborts the batch. Existing isolation tests cover only LM Studio failures ([tests 436-469](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/tests/test_syncer.py#L436-L469)). | +| Bundles `status` crashes when `lms` is broken | **Confirmed, bundles only.** Status conditionally calls `list_models()` with no guard ([77-99](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/service.py#L77-L99)); the adapter translates execution and JSON failures to `LMStudioError` ([66-110](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/lmstudio_adapter.py#L66-L110), [149-176](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/lmstudio_adapter.py#L149-L176)). `inventory=None` is already interpreted as unsynced/pending for projector records ([service 392-397](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/service.py#L392-L397)), so the proposed fallback fits the current model. Current status tests cover an empty inventory, not an inventory exception ([tests 430-461](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/tests/test_service.py#L430-L461)). | +| UTF-8 BOM drops state | **Confirmed on all three refs.** `_load_raw()` uses `encoding="utf-8"` and turns the resulting JSON decode error into `{}`/“Starting fresh”: [main 72-82](https://github.com/ensevengg/StudioLink/blob/221229ddd9df6176134a1da695fd82c0a21499cb/src/studiolink/state.py#L72-L82), [bundles 71-81](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/state.py#L71-L81). Sync then sees no records and re-imports. `utf-8-sig` on reads is the narrow correct fix; writes can remain UTF-8. No BOM regression test exists. | + +## Additional bundles merge blockers + +### Projector-first manifests abort the batch + +The adapter preserves the manifest's model/projector layer order +([adapter 133-162](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/ollama_adapter.py#L133-L162)), +and sync imports in that order. After *each* imported artifact it creates and +saves a partial record ([syncer 245-265](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L245-L265)). +But serialization unconditionally resolves `primary_artifact` +([models 211-217](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/models.py#L211-L217), +[240-248](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/models.py#L240-L248)). +A projector-only partial has no primary, so `next(...)` raises `StopIteration`; +the outer batch loop does not catch it. + +A temporary probe with layers `[projector C, model A]` produced artifact order +`['projector', 'model']`, imported only the projector, then raised +`StopIteration`. Normalize supported artifacts to model-first before import (or +defer partial persistence until a primary exists) and add both a standalone and +two-model batch regression. The current fixture's model-first order does not +establish a general ordering invariant. + +### Changed vision bundles are split across old and new repositories + +The bundles branch keys `import_user_repo` by *all* artifact digest prefixes +([models 120-129](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/models.py#L120-L129)). +On re-sync it reuses an artifact solely by `(role, digest)` and target existence +([syncer 146-188](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L146-L188)), +but imports a changed artifact into the newly computed repository +([190-217](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L190-L217)). +LM Studio pairing requires both files in one directory +([the branch's end-to-end validation](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/docs/research/multi-artifact-validation.md#L8-L13)), +and reconciliation tests the new primary path for vision +([374-397](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L374-L397)). + +A temporary-directory probe synced `(model=A, projector=C)`, changed only the +model to `B`, then synced again. The first sync was `synced`; the second imported +one file and returned `error: LM Studio did not index the projector bundle as +vision-capable`. State pointed to: + +```text +.../llava--bundle-bbbbbbbbbbbb-cccccccccccc/ +.../llava--bundle-aaaaaaaaaaaa-cccccccccccc/ +``` + +Retry reuses both split targets and repeats the same error. The inverse change is +more dangerous: with `(model=A, projector=C)` already installed, changing only +the projector to `D` reuses the old primary in the old `A-C` directory (where old +projector C remains), imports D alone into the new `A-D` directory, then confirms +vision against the *old* A-C inventory entry. State and `status` can therefore +say synced while LM Studio still pairs A with stale projector C. + +The fix should make reuse conditional on the target's parent being the current +destination, or re-import/re-link every required artifact into the new bundle +directory. Add tests for changing only the primary digest and only the projector +digest, asserting co-location and the exact inventory path used for confirmation. + +## Conditional platform risks + +### Linux protected hard links + +**Strongly substantiated; the kernel mechanism was reproduced in WSL, but not +against an installed Ollama service.** With `fs.protected_hardlinks=1`, a +hard-link attempt by an unprivileged user to a root-owned `0644` source was +denied, while the same probe succeeded after the source was changed to `0666`. +That isolates the expected ownership/write-permission mechanism; the probe host +did not have a real Ollama installation. Stabilize changes the Linux default +store to `/usr/share/ollama/.ollama/models` +([config 18-21](https://github.com/ensevengg/StudioLink/blob/3d5c60b1bc6b3927b1fb2be196d0fd87c591526c/src/studiolink/config.py#L18-L21)). +Ollama's official system-service instructions create and run as the dedicated +`ollama` user ([official Linux docs](https://github.com/ollama/ollama/blob/205a0426905d45f5ed0d120f3c671e3573e01b41/docs/linux.mdx#adding-ollama-as-a-startup-service-recommended)), +and Ollama explicitly chmods newly created blobs to `0644` +([official source](https://github.com/ollama/ollama/blob/205a0426905d45f5ed0d120f3c671e3573e01b41/manifest/layer.go#L52-L59)). +With `fs.protected_hardlinks=1`, Linux rejects links by users who neither own nor +have read/write access to the source +([kernel documentation](https://docs.kernel.org/admin-guide/sysctl/fs.html#protected-hardlinks)). + +StudioLink always first calls `os.link(blob, alias)`, including copy mode +([stabilize 208-238](https://github.com/ensevengg/StudioLink/blob/3d5c60b1bc6b3927b1fb2be196d0fd87c591526c/src/studiolink/syncer.py#L208-L238)). +The doctor check compares only `st_dev` values +([service 323-348](https://github.com/ensevengg/StudioLink/blob/3d5c60b1bc6b3927b1fb2be196d0fd87c591526c/src/studiolink/service.py#L323-L348)), +so a typical single-filesystem machine can pass while link creation returns +`EPERM`; copy staging can also fail earlier if the user cannot create +`.studiolink-imports` under the Ollama-owned store. An actual reversible +`os.link()` probe using a real blob and each relevant destination is materially +better than a permission-bit or volume-only check. The error should explain +ownership/`protected_hardlinks` and the supported remediation; a doctor warning +alone does not make the default hard-link workflow work. + +### Zero/unknown inode identifiers + +**The code defect is confirmed; the exact exFAT/FAT incidence was not reproduced.** +Every ref treats equal `(st_dev, st_ino)` as conclusive +([bundles 34-43](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L34-L43)). +Python only promises uniqueness when `st_ino` is nonzero +([official `stat_result` docs](https://docs.python.org/3/library/os.html#os.stat_result)). +Thus two distinct files reported as `(same device, 0)` are misclassified and a +wrong pre-existing alias can be reused. Return false if either identifier is +zero. Comparing `st_size` is useful as another mismatch check, but equal sizes +do not establish identity and therefore do not fully fix stale-weight reuse. + +## Design-level reliability gaps + +| Claim | Verdict and refinement | +|---|---| +| No fsync before replace | **Confirmed on all refs.** State is written with `Path.write_text()` and immediately replaced, with no file flush/fsync and no parent-directory fsync ([bundles 63-69](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/state.py#L63-L69)). Atomic namespace replacement is not a power-loss durability guarantee. A robust sequence is unique temp file -> flush/fsync file -> replace -> fsync parent where supported. | +| No locking/fixed temp | **Confirmed on all refs.** Every writer uses `state.json.tmp` ([bundles 67-69](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/state.py#L67-L69)) and read/modify/write operations have no inter-process lock. Concurrent commands can lose updates; the shared temp name also permits replace races and `FileNotFoundError`. Use a state-scoped inter-process lock around the full transaction and unique same-directory temp files. | +| Prune ignores copy staging | **Confirmed on both improvement refs.** Copy aliases live under `/.studiolink-imports` ([bundles 421-433](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L421-L433)), while prune scans only `config.import_staging_dir` ([service 176-213](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/service.py#L176-L213)). A process crash or cleanup failure can therefore leave a hard link that pins bytes indefinitely. | +| Dry-run has filesystem side effects | **Refine.** It really does create a staging directory and hard link before invoking `lms --dry-run`; it is not intrinsically read-only ([bundles 194-227](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L194-L227)). However, successful stabilize/bundles runs unlink the alias *and rmdir the staging leaf* ([399-419](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L399-L419)); because a real blob implies `ollama_models_dir` already exists, successful copy dry-runs do not normally leave parent directories. Persistent empty `.studiolink-imports` is an error/crash path (notably `mkdir` succeeds then `os.link` fails), not the normal success path. Main does leave its state staging directory after a successful dry-run because it unlinks only the file ([main 151-159](https://github.com/ensevengg/StudioLink/blob/221229ddd9df6176134a1da695fd82c0a21499cb/src/studiolink/syncer.py#L151-L159)). | +| Missing source blob: text skipped, vision errors | **Confirmed in bundles.** A synced text model returns before the stale check, while projector models deliberately bypass that return and reach the stale error ([syncer 104-126](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L104-L126)). Choose and test one policy; reporting stale consistently is least surprising. | +| Old LM Studio imports are orphaned | **Confirmed.** Filenames include digest prefixes ([models 139-149](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/models.py#L139-L149)); vision repositories include all digest prefixes ([120-129](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/models.py#L120-L129)). Prune removes staging aliases and state only, never LM Studio targets. Text re-pulls leave old digest files; vision re-pulls leave old bundle directories. Deletion needs an explicit ownership/safety policy because those directories are user-visible LM Studio data. | +| Upgrade prerelease comparison | **Confirmed, unchanged.** `_version_tuple()` strips non-digits per dotted component, so `0.2.1rc1` becomes `(0, 2, 11)` ([CLI 320-325](https://github.com/ensevengg/StudioLink/blob/221229ddd9df6176134a1da695fd82c0a21499cb/src/studiolink/cli.py#L320-L325)); the test codifies the faulty behavior ([tests 240-241](https://github.com/ensevengg/StudioLink/blob/221229ddd9df6176134a1da695fd82c0a21499cb/tests/test_cli.py#L240-L241)). Use standards-aware version parsing. | +| Pip upgrade unpinned | **Confirmed, unchanged.** The command installs unpinned `studiolink --upgrade` after separately reading a reported version ([CLI 272-301](https://github.com/ensevengg/StudioLink/blob/221229ddd9df6176134a1da695fd82c0a21499cb/src/studiolink/cli.py#L272-L301)). Pin the fetched version if the success message is meant to be authoritative. | +| ANSI/control-sequence spoofing | **Confirmed, print-only, unchanged.** Canonical names derived from manifest path components are printed directly by scan/sync/status ([CLI 183-188](https://github.com/ensevengg/StudioLink/blob/221229ddd9df6176134a1da695fd82c0a21499cb/src/studiolink/cli.py#L183-L188), [206-208](https://github.com/ensevengg/StudioLink/blob/221229ddd9df6176134a1da695fd82c0a21499cb/src/studiolink/cli.py#L206-L208), [223-232](https://github.com/ensevengg/StudioLink/blob/221229ddd9df6176134a1da695fd82c0a21499cb/src/studiolink/cli.py#L223-L232)). JSON output remains machine-safe encoding; sanitize controls in terminal rendering. | + +## “Probed safe” claims + +| Claim | Verdict and scope | +|---|---| +| Prune is symlink-safe | **Destructive action is safe, wording needs nuance.** `is_file()`, `same_file()`, and `stat()` follow a planted symlink for metadata, but deletion is `path.unlink()`, which removes the link entry rather than its target ([bundles service 179-204](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/service.py#L179-L204)). So “target never deleted” is accurate; “never followed” is not. An expected-name symlink to the current blob can also satisfy `same_file()` and be preserved rather than unlinked. | +| `--direct` never deletes the Ollama blob | **Confirmed.** Direct mode uses the blob as `alias_path` ([bundles 190-192](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L190-L192)), and cleanup explicitly excludes direct mode ([399-413](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L399-L413)). | +| Ctrl-C cleans temporary aliases | **Confirmed for ordinary Python interruption.** Import is inside `try/finally`, so `KeyboardInterrupt` runs cleanup before propagating ([bundles 209-227](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L209-L227)). This cannot cover hard process termination or power loss. | +| Concurrent alias-creation race is non-corrupting | **Confirmed for the absent-alias race.** One `os.link` wins; the other receives an `OSError` translated to `RuntimeError` ([bundles 459-464](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L459-L464)). It fails one model rather than overwriting content. This does not make state writes concurrency-safe. | +| Artifact ordering is consistent | **Refuted as a safety claim.** The adapter and syncer consistently preserve *manifest* order ([adapter 133-162](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/ollama_adapter.py#L133-L162), [syncer 158-188](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/syncer.py#L158-L188)), but they do not enforce model-first. A projector-first manifest crashes while saving the first partial record, as detailed above. Normalize semantic order rather than relying on the current fixture. | +| Schema v2 downgrades cleanly to 0.2.0 | **No-crash compatibility confirmed, but it is lossy.** Bundles keeps legacy primary fields at the record top level ([models 240-260](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/models.py#L240-L260)), and 0.2.0 ignores extra fields while parsing ([main models 117-129](https://github.com/ensevengg/StudioLink/blob/221229ddd9df6176134a1da695fd82c0a21499cb/src/studiolink/models.py#L117-L129)). But the next 0.2.0 bulk `save()` serializes legacy records and schema 1, discarding projector/vision metadata ([main state 44-70](https://github.com/ensevengg/StudioLink/blob/221229ddd9df6176134a1da695fd82c0a21499cb/src/studiolink/state.py#L44-L70)). Safe means “does not crash,” not round-trip preservation. | +| Corrupt manifest JSON protects aliases | **Confirmed on stabilize and bundles, not main.** Invalid JSON becomes an `INVALID` model rather than disappearing ([bundles adapter 116-126](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/ollama_adapter.py#L116-L126)); prune protects aliases for every unready model with a record ([service 163-183](https://github.com/ensevengg/StudioLink/blob/1090dbcb80436e3a0a4be4d7bc3db95b8b853acf/src/studiolink/service.py#L163-L183)). Main has no `unready_names` protection and can remove that alias. | + +## Test additions worth requiring with the fixes + +- Two-model batches where the first alias hits `PermissionError` from `mkdir()` + and from stale-alias `unlink()`; the second model must still sync. +- Vision `status()` where `list_models()` raises `LMStudioError`; output must be + pending and the command successful. +- A BOM-prefixed state file containing a real record, not just empty JSON. +- Projector-first valid manifests, including a two-model batch; the model must be + imported first or no invalid projector-only partial may be saved. +- Vision re-pulls changing only the model digest and only the projector digest; + all imported paths must share the current bundle parent and reconciliation + must succeed. +- Mocked zero inode/file identifiers for two distinct files; `same_file()` must + return false even when their sizes match. +- Copy dry-run/link-failure cleanup and prune coverage for + `.studiolink-imports` leftovers. diff --git a/src/studiolink/__init__.py b/src/studiolink/__init__.py index 8fe341e..70e1644 100644 --- a/src/studiolink/__init__.py +++ b/src/studiolink/__init__.py @@ -2,4 +2,4 @@ __all__ = ["__version__"] -__version__ = "0.3.0" +__version__ = "0.3.1"