Support graceful script isolation fallback when filesystem does not support locking

[rhysparry]

Script isolation lock directory fallback for filesystems without full locking support

Motivation

Calamari's script isolation mechanism uses filesystem file locks to serialise or permit concurrent script execution. The lock file has historically been written into the Tentacle working directory (TentacleHome).

On some deployment targets the working directory resides on a filesystem that does not fully support file locking semantics — NFS mounts, certain FUSE-based filesystems, and some SMB shares being common examples. On these filesystems, lock acquisition either silently succeeds when it should block, or throws errors that are not meaningful to the caller. The result is that script isolation is silently broken: scripts that should be mutually exclusive may run concurrently, or scripts that should run in parallel are incorrectly serialised (or errored).

This PR introduces runtime filesystem lock capability detection and an automatic fallback to a temporary directory that does support locking, so that script isolation remains as strong as the host OS allows — without requiring any manual configuration from operators.

---

What changed

New abstractions

File	Purpose
ScriptIsolation/LockCapability.cs	Enum: Unknown, Unsupported, ExclusiveOnly, Supported
ScriptIsolation/LockFile.cs	Value type pairing a LockDirectory with a specific file path
ScriptIsolation/LockDirectory.cs	Value type capturing a directory path + its detected LockCapability; hosts the GetLockDirectory fallback algorithm
ScriptIsolation/CachedDriveInfo.cs	Snapshot of a drive/mount's format, type, and detected lock capability; performs live 4-probe detection
ScriptIsolation/MountedDrives.cs	Enumerates OS mounts; maps a path to its best-matching drive via longest-prefix matching with symlink resolution
ScriptIsolation/TemporaryDirectoryFallback.cs	Platform-aware enumeration of candidate temp directories to fall back to
ScriptIsolation/IFileLockService.cs + FileLockService.cs	Injectable abstraction over CreateDirectory, AcquireLock, and GetFallbackTemporaryDirectories — enables hermetic unit tests
ScriptIsolation/IPathResolutionService.cs + DefaultPathResolutionService.cs + PathResolutionServiceExtensions.cs	Injectable abstraction and extension over Path.GetFullPath and symlink resolution — necessary to correctly match paths on macOS where /tmp → /private/tmp

Modified files

File	Change
ScriptIsolation/Isolation.cs	Adds PrepareLockOptions and ResolveLockOptions — the degradation policy when a directory with full lock support cannot be found
ScriptIsolation/LockOptions.cs	Delegates IsFullySupported / IsSupported / Supports(LockType) to LockFile; adds FromScriptIsolationOptionsOrNull path through LockDirectory.GetLockDirectory
ScriptIsolation/FileLock.cs	Minor: adapts to LockFile record shape
Plumbing/Commands/CommonOptions.cs	Adds scriptIsolationSharedFallback CLI option and PromoteToExclusiveLockWhenSharedLockUnavailable property

---

Decision trees

Phase 1 — Lock directory selection (LockDirectory.GetLockDirectory)

flowchart TD
    Start([Candidate path]) --> MapDrive[Map candidate to its drive]
    MapDrive --> CandidateKnown{Candidate LockSupport?}
    CandidateKnown -- Supported --> ReturnCandidate([Return candidate\nLockCapability.Supported])
    CandidateKnown -- Unknown or Unsupported --> ProbeCandidateDrive

    ProbeCandidateDrive[DetectLockSupport on candidate drive]
    ProbeCandidateDrive --> CandidateProbeResult{Candidate\ndetected capability?}
    CandidateProbeResult -- Supported --> ReturnCandidate2([Return candidate\nLockCapability.Supported])
    CandidateProbeResult -- ExclusiveOnly or Unsupported or Unknown --> IterateFallbacks

    IterateFallbacks[Iterate temp fallback directories]
    IterateFallbacks --> HasTemp{More temp\ncandidates?}
    HasTemp -- No --> AnyExclusiveTemp
    HasTemp -- Yes --> MapTempDrive[Map temp dir to its drive]
    MapTempDrive --> ProbeTemp[DetectLockSupport on temp drive]
    ProbeTemp --> TempResult{Temp capability?}
    TempResult -- Supported --> ReturnTemp([Return temp dir\nLockCapability.Supported])
    TempResult -- ExclusiveOnly --> RecordFirstExclusive[Record as best-so-far\nif none recorded]
    TempResult -- Unsupported or Unknown --> HasTemp

    AnyExclusiveTemp{Any ExclusiveOnly\ntemp found?}
    AnyExclusiveTemp -- Yes, and candidate is ExclusiveOnly --> ReturnCandidate3([Return candidate\nLockCapability.ExclusiveOnly])
    AnyExclusiveTemp -- Yes, and candidate is Unsupported/Unknown --> ReturnBestTemp([Return best temp dir\nLockCapability.ExclusiveOnly])
    AnyExclusiveTemp -- No --> ReturnUnsupported([Return candidate\nLockCapability.Unsupported])

Loading

Key design choice: The candidate drive is probed before iterating temp directories. If the candidate detects as Supported it is returned immediately without ever consulting temp paths. When both the candidate and the best temp are ExclusiveOnly, the candidate is still preferred — using a temp directory offers no additional isolation benefit and introduces unnecessary path indirection. The temp is only used when it genuinely improves on the candidate.

---

Phase 2 — Lock options resolution (Isolation.ResolveLockOptions)

flowchart TD
    Start([LockOptions\nType + LockDirectory capability]) --> FullySupported{IsFullySupported?\nShared + Exclusive work\ncorrectly}

    FullySupported -- Yes --> ReturnOriginal([Use requested lock\nNo warning])
    FullySupported -- No --> IsSupported{IsSupported?\nRequested LockType\nis supported}

    IsSupported -- Yes, Exclusive requested --> PromoteFlag1{promoteToExclusiveLock?}
    PromoteFlag1 -- false --> ReturnExclusiveWarn([Use Exclusive lock\nWarn: shared scripts\nmay run concurrently])
    PromoteFlag1 -- true --> ReturnExclusiveNoWarn([Use Exclusive lock\nNo warning])

    IsSupported -- No, Shared requested\non ExclusiveOnly dir --> ExclusiveAvail{Supports\nExclusive?}
    ExclusiveAvail -- Yes --> PromoteFlag2{promoteToExclusiveLock?}
    PromoteFlag2 -- true --> PromoteToExclusive([Promote to Exclusive lock\nWarn: shared unavailable])
    PromoteFlag2 -- false --> NoLockShared([No lock acquired\nWarn: shared unavailable])

    ExclusiveAvail -- No --> NoLockUnsupported([No lock acquired\nWarn: no isolation available])

Loading

The promoteToExclusiveLock flag is controlled by the new scriptIsolationSharedFallback CLI option. It defaults to Exclusive (promote), which is the conservative choice — a script running with NoIsolation that cannot get a shared lock is promoted to exclusive rather than running with zero isolation. Pass Skip to disable promotion.

---

Areas especially relevant for review

CachedDriveInfo.DetectLockSupport — live filesystem probing

Four sequential probe operations are run against a temporary .tmp file in the lock directory to empirically verify lock semantics. The probes use TimeSpan.Zero timeout so they fail fast if the filesystem rejects the operation. The test file is always cleaned up in a finally block. Any exception during probing is treated conservatively as Unsupported. Reviewers should check whether the probe-file cleanup is sufficient and whether the TimeSpan.Zero strategy is reliable across all target filesystems.

PathResolutionServiceExtensions.ResolvePath — ancestor-walk symlink resolution

The ancestor-walk is necessary because a lock directory might not exist yet at the time of drive matching (e.g. first ever run). The algorithm peels off non-existent path segments from the end, resolves the nearest existing ancestor's symlink, then re-attaches the original tail. This is especially important on macOS where Path.GetFullPath("/tmp/x") returns /tmp/x but the drive is mounted at /private/tmp. All five GetFullPath exception types are caught and cause the input to be returned unchanged.

MountedDrives.GetAssociatedDrive — longest-prefix mount matching

The match is done on the symlink-resolved path, with platform-appropriate case sensitivity (OrdinalIgnoreCase on Windows/macOS, Ordinal on Linux). Reviewers should verify the StartsWith prefix check correctly handles the case where the drive root IS the path (no trailing separator needed since RootDirectory.FullName ends with a separator on all .NET platforms).

TemporaryDirectoryFallback.GetCandidates — platform-specific candidate order

On Linux, /dev/shm is included as a candidate after /tmp. tmpfs-backed /dev/shm is one of the most reliably lock-capable paths on systems where the working directory is NFS-mounted. The candidates are namespace-scoped using the last segment of the candidate path to prevent cross-deployment lock file collisions.

CommonOptions.ScriptIsolationOptions — new CLI parameter

The scriptIsolationSharedFallback option is additive and defaults to the safe value (Exclusive). Callers that do not pass the option get PromoteToExclusiveLockWhenSharedLockUnavailable = true.

---

Test coverage

LockDirectoryFixture — 1,063 lines, all groups tagged RunOnceOnWindowsAndLinux

All tests in this fixture use injected fakes (FakeLockService, FakePathResolutionService) and run entirely in-memory — no real filesystem access.

Group	What is tested
A — LockDirectory.Supports	All LockCapability × LockType combinations (6 cases)
B — CachedDriveInfo.LockSupport static	Known-good formats (apfs, ntfs, ext4, etc.) → Supported; network drives → Unknown; DetectedLockSupport overrides (13 cases)
C — CachedDriveInfo.DetectLockSupport	All four live-probe outcomes: Unsupported, three variants of ExclusiveOnly (broken shared, broken exclusive-blocks-shared, broken shared-blocks-exclusive), and Supported (4 cases)
D — LockDirectory.GetLockDirectory	9 scenario tests: candidate already Supported; candidate Unknown → detected Supported before temps inspected (candidate preferred over a Supported temp); candidate Unsupported → Supported temp used; candidate Unknown → detected Supported before ExclusiveOnly temps inspected; both candidate and temp are ExclusiveOnly → stay on candidate; temp ExclusiveOnly + candidate Unsupported → use temp; nothing works → Unsupported; empty drive table; temp with no associated drive skipped; first Supported temp wins when multiple candidates
E — MountedDrives.GetAssociatedDrive	7 tests: symlink resolution before matching; case-sensitive rejection; case-insensitive match; normalisation; longest-prefix wins (POSIX); DirectoryNotFoundException when no match; ancestor symlink resolved when child doesn't exist
F — DefaultPathResolutionService	3 real-filesystem integration tests (2 Unix-only): existing symlink, non-existent child under symlink ancestor, fully non-existent path
G — PathResolutionServiceExtensions.ResolvePath	9 tests: all GetFullPath exception types caught (5 parameterised), non-existent path, existing non-symlink, symlink followed, ancestor symlink + tail re-attached

IsolationFixture — extended with ResolveLockOptions unit tests

Tests added	What is tested
6 new unit tests	All branches of Isolation.ResolveLockOptions: Supported → original returned; ExclusiveOnly + Exclusive + promote=false → warning; ExclusiveOnly + Exclusive + promote=true → no warning; ExclusiveOnly + Shared + promote=true → promoted to Exclusive + warning; ExclusiveOnly + Shared + promote=false → null + warning; Unsupported + any request → null + warning

LockOptionsFixture

FromScriptIsolationOptionsOrNull_BuildsCorrectLockFileName updated to assert on LockFile.File.Name rather than the full path (since the lock directory is now resolved dynamically rather than being a fixed subdirectory of TentacleHome).

What is not covered by unit tests

• The end-to-end path through LockDirectory.GetLockDirectory against a real NFS or FUSE mount. The live probing logic is verified against a real (local) filesystem via IsolationFixture and Group F in LockDirectoryFixture, but the specific scenario of a network filesystem detecting as Unsupported and falling back to /tmp is exercised only through the fake in Group D.
• Windows-specific temp directory candidates (%LOCALAPPDATA%\Calamari, %TEMP%) are not covered by integration tests.

⚠️ Does this change require a corresponding Server Change?
⚠️ If so - please add a "Requires Server Change" label to this PR!

[gb-8]

I've had a scan of the PR and while the approach seems good, and the test coverage seems good, I did find the code hard to follow.
I had to really work to follow the logic through, and see exactly where the determination of locking support was performed.
The logic is all hidden inside things that don't make it clear what they are doing.
Once you are happy that the functionality is all correct, then I'd like to pair on seeing if we can refactor to make it more readable and understandable.

[gb-8]

Claude reckons we are using these values wrongly, since when we are unable to determine whether locking is supported, we fallback on Unsupported when it thinks it technically should be Unknown.

I think we are actually using Unknown to mean Untested, so could rename it as such.

[gb-8]

I think it is good practice to set the default value explicitly to 0.

[gb-8]

I wonder if we should actually preference /dev/shm over other locations, as it has the nice behaviour that locks will automatically get cleaned up on reboot.