feat: multi-GPU parallel session execution

[lstein]

Summary

This PR adds multi-GPU parallel generation: on a machine with more than one GPU, InvokeAI runs several generation sessions concurrently — one per GPU — instead of draining the queue one job at a time. Jobs are distributed fairly across users so a single user's large batch can't monopolize every GPU while others wait.

It's controlled by a new generation_devices config setting (defaults to auto = use every available CUDA GPU). Setting it to a single device, or leaving CUDA out of the picture, preserves the previous serial behavior exactly. The choice of GPUs can also be controlled via a new section of the Settings dialogue (restart required to take effect).

Demo (turn on the sound!)

invoke-mgpu.mp4

How it works — the change is built around five small backend seams plus a frontend update, rather than per-node edits:

• Device context (invokeai/backend/util/devices.py): a thread-local set/get/clear_session_device on TorchDevice; choose_torch_device() consults it first. This is the lynchpin — the ~79 existing call sites resolve to the worker's GPU with no per-node changes.
• Per-device model caches (model_manager_default, model_load_default): one ModelCache per device, resolved by the current thread's device, with fan-out for clear/drop/shutdown. Model construction is serialized against VRAM moves to prevent meta-device corruption. A single global RAM budget is shared across the per-device caches, and identical CPU weights are deduplicated across devices (see RAM management below).
• Atomic dequeue (session_queue_sqlite.dequeue): a lock makes select+claim atomic so concurrent workers never grab the same queue item.
• Worker pool (session_processor_default): one _SessionWorker per device, each pinning torch.cuda.set_device + the session device, with its own runner and cancel event; cancellation is routed per item. Profiling is disabled when more than one worker is active.
• Concurrency hardening: added a Lock to ObjectSerializerForwardCache and made DiskImageFileStorage thread-safe for parallel sessions.

Frontend: during parallel generation the progress display stacks one progress bar per active session (each disappears as its session finishes), and the image viewer tiles per-session progress previews when ≥2 sessions are active.

Idle-GPU text-encoder offload

When more than one generation device is configured and a GPU is idle, a session's text/prompt encoder runs on the idle GPU instead of the one running its denoise pipeline. This avoids evicting the denoise model from VRAM to make room for the encoder, and lets a cached encoder be reused across generations. Under full load (no idle GPU) behavior is unchanged. Controlled by offload_text_encoders_to_idle_gpus (default on); inspired by #9310.

• Device-pool arbiter (backend/util/device_pool.py): GENERATION_DEVICE_POOL gives each generation device one exclusive-use lock. A native session blocking-acquires its own device's lock for the whole run; an encoder node try-borrows an idle device's lock for the duration of that node. A borrowed encoder and a native session are therefore mutually exclusive on a GPU — preventing the shared-encoder corruption that produced garbled images — and the design is deadlock-free (borrows are non-blocking; a session only ever blocks on its own device).
• Opt-in marker: nodes declare support via @invocation(idle_gpu_offloadable=True), mirroring the existing bottleneck ClassVar. Applied to the text/prompt-encoder nodes (compel + sdxl/refiner, flux, sd3, qwen-image, anima, cogview4, flux2 klein, z-image, flux_redux). The runner re-pins the worker thread to the borrowed device for the node; conditioning is stored on the CPU so the denoiser picks it up on its own GPU afterward.

RAM management for parallel sessions

Running N sessions in parallel multiplies memory pressure, so this PR also makes the model cache parallel-aware:

• Shared global RAM budget, clamped to a safe fraction of system RAM, so summing per-device cache sizes across GPUs can't claim ~N× RAM and drive the box into swap.
• Cross-device weight de-duplication: when a second GPU loads a model another GPU already holds, it adopts the resident CPU weights (a meta-weight structural clone + load_state_dict(assign=True)) instead of re-reading from disk and materializing a second copy. This is loader-agnostic and now also covers GGUF models — GGMLTensor doesn't implement aten.empty_like, which previously made the largest quantized models (e.g. a Q8_0 transformer) silently re-load on every device and spike RAM; the adopted GGMLTensor shares the quantized storage, so it's one copy across devices.

Generation Devices settings refinements

A few small fixes to the Generation Devices selector and its logging:

• Stable device numbering: the disambiguating #N suffix on identically-named GPUs is now tied to each device's cuda index (its position in the full available-device set) rather than its position in the possibly-filtered generation_devices list. Previously, disabling e.g. cuda:1 renumbered the survivors in the backend startup log (cuda:2 became #2), disagreeing with the frontend, which always labels over the full set. Now both stay consistent — cuda:2 remains #3.
• Restart reminder: reworded the Settings caption to "Restart InvokeAI for changes to take effect." and flash that same warning as a toast on every successful change, since generation_devices only takes effect after a restart.

Related Issues / Discussions

• Builds conceptually alongside [feat] Round robin job scheduling in multiuser mode #9086 (round-robin per-user dequeue). This PR does not depend on [feat] Round robin job scheduling in multiuser mode #9086 being merged — the atomic claim works on the current FIFO dequeue, and [feat] Round robin job scheduling in multiuser mode #9086's round-robin CTE can slot in cleanly when it lands.
• Idle-GPU text-encoder offload was inspired by Add split GPU text encoder cache #9310 (now folded into this PR; supersedes the earlier private-repo prototype).

QA Instructions

On a multi-GPU machine:

• With default config (generation_devices: auto), enqueue a batch larger than the GPU count and confirm multiple sessions run simultaneously (one per GPU), with stacked progress bars and tiled previews in the viewer.
• Set generation_devices: [cuda:0] and confirm generation runs serially, exactly as before this PR.
• Set generation_devices: [cuda:0, cuda:2] and confirm only those devices are used.
• Cancel an in-flight item and confirm only that session stops.
• On a single-GPU / CPU / MPS machine, confirm auto resolves to the one best device and behavior is unchanged.
• Idle-GPU offload: with ≥2 GPUs and a single running session, confirm the text encoder runs on a different (idle) GPU than the denoiser (debug log: Running ... on idle device ...), and that the denoise model is not evicted to load the encoder. Set offload_text_encoders_to_idle_gpus: false and confirm the encoder runs on the session's own GPU.
• Parallel RAM: run several parallel sessions with the same model (ideally a GGUF/quantized one) and confirm process RAM stays bounded — the transformer/text-encoder show an Adopted shared CPU weights ... log on the second device rather than a second disk load.

New automated tests cover device routing (test_model_load_device_routing.py), dequeue concurrency (test_session_queue_dequeue_concurrency.py), device resolution (test_devices.py), the device-pool lock semantics and offload mutual-exclusion (test_device_pool.py, test_encoder_offload.py), and cross-device weight adoption incl. GGUF (test_shared_weight_adoption.py).

Merge Plan

Standard merge. No DB schema or redux migrations. Touches the session processor and model cache, so worth a careful look from those areas' owners.

The idle-GPU text-encoder offload (originally prototyped as a follow-on PR) is now included in this branch, along with the cross-device GGUF weight de-duplication that keeps parallel-session RAM bounded.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• ❗Changes to a redux slice have a corresponding migration — N/A, no slice changes
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[JPPhoto]

• Stale shared CPU weights can still survive a load-affecting model settings change. Affected paths: invokeai/app/api/routers/model_manager.py:445, invokeai/backend/model_manager/load/model_cache/model_cache.py:1144, invokeai/backend/model_manager/load/model_cache/shared_cpu_weights.py:59, invokeai/backend/model_manager/load/model_cache/cached_model/cached_model_only_full_load.py:52, invokeai/backend/model_manager/load/model_cache/cached_model/cached_model_with_partial_load.py:61. drop_model() only marks locked entries stale, so their shared-store reference stays live until unlock. Another device cache can reload the same key before that unlock and adopt the old canonical tensors. To expose this issue, add a test that locks a shared cached model, calls drop_model() across caches, then reloads the same key in another cache with different weights/settings and asserts the new entry cannot adopt the old canonical state dict.

• Runtime config accepts invalid generation_devices values at the API boundary. Affected paths: invokeai/app/api/routers/app_info.py:151, invokeai/app/services/config/config_default.py:276, invokeai/backend/util/devices.py:218. The route validator only checks the string pattern, so it accepts [] until later config validation and accepts syntactically valid but unavailable devices like cuda:99 until later device resolution/startup. To expose this issue, add route tests for [] and an out-of-range CUDA index under mocked CUDA availability/device count, asserting a 422 and no config-file mutation.

• Single-user mode is documented/configured as FIFO, but default device affinity reorders it. Affected paths: invokeai/app/services/config/config_default.py:221, invokeai/app/services/session_queue/session_queue_sqlite.py:304, invokeai/app/services/session_queue/session_queue_sqlite.py:320, tests/app/services/session_queue/test_session_queue_dequeue.py:390. The config says single-user mode always uses FIFO, but default session_queue_mode="round_robin" still enables affinity, allowing a later warm item to jump ahead of an older cold item. Either preserve strict FIFO in single-user mode or update config/docs to say same-priority jobs may be reordered for device affinity.

• Multi-GPU docs still describe invalid and stale behavior. Affected path: docs/src/content/docs/configuration/invokeai-yaml.mdx:132, docs/src/content/docs/configuration/invokeai-yaml.mdx:144. The docs say generation_devices: [] is valid, but config validation rejects empty lists. They also say model weights are duplicated in system RAM per active GPU, while this PR adds shared CPU-weight deduplication. Update the docs to match the implemented behavior.

• Cache stats still report only the API thread's default cache. Affected paths: invokeai/app/api/routers/model_manager.py:1298, invokeai/app/services/model_load/model_load_default.py:61. get_stats() returns model_manager.load.ram_cache.stats, which resolves to the default cache for API request threads, while PR 9263 now has one cache per generation device. To expose this issue, add a test with distinct stats on two ram_caches and assert the endpoint returns aggregate or per-device stats instead of only the default cache.

• Open question: should queue priority remain global in round-robin mode? ROUND_ROBIN_DEQUEUE_QUERY chooses each user's best pending item, then orders users by least-recently served. That means a recently served user's high-priority/prepended job can wait behind another user's lower-priority job. If that is intentional fairness behavior, document it; otherwise the query needs another priority tier above user rotation.

[JPPhoto]

See prior comments in this PR.

[lstein]

@JPPhoto Thanks for the thorough pass — all five findings were real. Everything is addressed in aee12c3:

Stale shared CPU weights surviving a settings change — Confirmed. drop_model() now also calls a new SharedCpuWeightsStore.invalidate(model_key), which forgets the canonical entries (and shells) for the model and all its submodels, so no future acquire()/peek() can adopt them even while a locked, stale-marked entry still aliases them. Two supporting changes: release() is now identity-checked (a stale holder releasing after a new canonical was registered under the same key is a no-op rather than a decrement of the new entry's refcount), and update_model_record holds MODEL_LOAD_LOCK.write_lock() across the multi-cache drop (via asyncio.to_thread so the event loop isn't blocked) — that closes the narrower race where an in-flight load peeks the old canonical before the drop and re-registers it after. Added the test you described (test_settings_change_rebuild_does_not_adopt_stale_shared_weights: two caches, one locked, drop on both, rebuild in the unlocked cache, assert fresh canonical + identity-safe release on unlock) plus store-level unit tests for invalidate() and mismatched release.

Runtime config accepts invalid generation_devices — Confirmed. Empty lists are now rejected by the request model (422 instead of the eventual 500 from config validation), and the route resolves the requested devices through TorchDevice.get_generation_devices() — the same code the startup path uses — before anything is mutated or persisted, so cuda:99 on a 2-GPU box gets a 422 with the same message startup would have produced. Route tests cover [], out-of-range CUDA, and CUDA-unavailable under mocked device counts, asserting 422 and unchanged config.

Single-user FIFO vs. device affinity — The affinity reordering in single-user multi-GPU mode is intentional (a single user mixing models across 2 GPUs is exactly the thrash case that motivated it), so I took the document-it option: the session_queue_mode description no longer claims "FIFO is always used in single-user mode"; it now says jobs are served in submission order either way, except that on multi-GPU systems the default round_robin allows same-priority jobs to be reordered so a freed GPU prefers models it already has loaded, and that explicit FIFO disables the reordering. The multi-GPU docs got a matching note.

Stale multi-GPU docs — Fixed. The generation_devices: [] row is removed from the table (with an explicit "the list may not be empty" note), and the RAM note now describes the shared CPU-weight behavior (one RAM copy per model regardless of GPU count) instead of per-device duplication. settings.json regenerated.

Cache stats endpoint — Confirmed. /v2/models/stats now aggregates across ram_caches (deduplicating by cache object, since the default cache appears under its own device key), summing counters and merging loaded_model_sizes; single-cache installs return the same object as before. Tests cover aggregation, duplicate-object dedup, and the all-None → null case.

Open question (global priority vs. user rotation) — That ordering comes verbatim from #9086 on main (ROUND_ROBIN_DEQUEUE_QUERY is unchanged by this PR, which merely merged it), so I've treated it as intentional fairness-first design rather than changing it here: user rotation outranks priority across users; priority orders jobs within each user's queue. I've documented that in the session_queue_mode description. If we decide priority should form a tier above rotation, I'd rather that change land on main independently of this PR since it affects single-GPU multi-user installs equally.

[lstein]

@JPPhoto The fixes described above are committed and pushed (aee12c3) — this is ready for re-review whenever you have a chance. Thanks again for the careful pass.

[JPPhoto]

@lstein More:

• invokeai/backend/model_manager/load/model_cache/shared_cpu_weights.py:131 and invokeai/backend/model_manager/load/model_cache/ram_budget.py:55: invalidation undercounts RAM while stale models remain locked. invalidate() deletes the canonical entry immediately, so total_bytes_in_use() stops counting its tensors even though locked cache entries still retain them. If another device rebuilds the model before those entries unlock, both the retired and replacement weights are resident, but RamBudget counts only the replacement. Memory admission can therefore exceed max_cache_ram_gb, recreating the RAM spike or process OOM that the global budget is intended to prevent. To expose this issue, add a test that locks a shared model, invalidates it, registers replacement weights, and asserts budget usage includes both copies until the stale holder releases its reference.

• invokeai/app/api/routers/model_manager.py:1327 and invokeai/backend/model_manager/load/model_cache/model_cache.py:367: the new cache-stat aggregation multiplies system-wide values by the number of device caches. Every cache receives the same global RamBudget, so each CacheStats.cache_size is set to the same global max_bytes; _get_ram_in_use() likewise returns the shared global usage used for high_watermark. Summing these fields reports an N-GPU system's cache capacity and high-water usage approximately N times too large. The added test uses unrelated synthetic capacities and therefore codifies the incorrect behavior instead of representing the shared-budget setup. To expose this issue, add a test with two distinct caches attached to the same RamBudget and assert cache_size equals the single global limit and high_watermark is not multiplied.

• invokeai/app/api/routers/app_info.py:217 and invokeai/backend/util/devices.py:193: runtime validation still accepts an unavailable MPS device. TorchDevice.get_generation_devices() validates CUDA availability and indices but performs no equivalent check for device.type == "mps". An admin on Linux or an unsupported macOS environment can persist generation_devices: ["mps"]; startup then creates a worker and cache targeting an unusable device, deferring failure until tensor operations begin. This also contradicts the route comment claiming it prevents persistence of devices that do not exist. To expose this issue, add route and device-resolution tests with torch.backends.mps.is_available() returning false, submit ["mps"], and assert a 422 with no runtime or persisted config mutation.

• The database-facing tests changed by this PR use the shared InvokeAIAppConfig(use_memory_db=True) fixture or explicit sqlite3.connect(":memory:") connections, including tests/app/services/session_queue/test_session_queue_dequeue.py, tests/app/services/session_queue/test_session_queue_dequeue_concurrency.py, tests/app/services/session_queue/test_session_queue_multigpu_cancel.py, and tests/app/services/shared/sqlite_migrator/migrations/test_migration_2026_07_01_add_session_queue_device.py. No file-backed database dependency is apparent in those tests.

[lstein]

@JPPhoto Your July 12 findings are all addressed in e69b67e (pushed, CI green) — ready for re-review whenever you have a chance:

• Invalidation undercounting RAM while stale models remain locked — invalidate() now moves still-referenced entries to a retired list instead of dropping them from accounting; total_bytes_in_use() (and therefore RamBudget) keeps counting them until the last holder releases. release() matches retired entries by state-dict identity so a stale holder can't decrement a replacement canonical. Both tests you suggested are added (single holder, and multiple holders with a replacement registered — asserting both copies are counted until the last release).
• Stats aggregation multiplying system-wide values by N — /models/stats now takes the max of cache_size and high_watermark across per-device caches (they share one global RamBudget) and still sums event counters. The synthetic-capacity test was rewritten as you suggested: two caches attached to the same RamBudget, asserting cache_size equals the single global limit and high_watermark is not multiplied.
• Runtime validation accepting an unavailable MPS device — get_generation_devices() now rejects mps when torch.backends.mps.is_available() is false, so the route 422s with no config mutation. Route- and device-level tests added with MPS unavailable.

And you were right on the fourth point — the changed DB-facing tests have no file-backed dependency, so they stay on memory DBs.

---

Separately: to save you a round, I ran a deep self-review of the whole branch and found the issues below (each independently verified against the code). I'm intentionally not fixing them yet so your re-review of the current head isn't chasing a moving target — they'll land as a follow-up batch (each with an exposure test) once you've had a look. Listing them now for transparency:

1. invokeai/backend/patches/layer_patcher.py:214 — FLUX Control LoRA shape expansion creates an nn.Parameter via setattr with no lock held; if another worker's cold load holds MODEL_LOAD_LOCK.write_lock() with accelerate's process-global register_parameter→meta patch active, the expanded weight lands on the meta device → "Cannot copy out of meta tensor" / corrupted module. Planned fix: take MODEL_LOAD_LOCK.read_lock() around patch application.
2. invokeai/app/invocations/flux_redux.py + invokeai/app/invocations/flux_denoise.py:602 — FluxRedux is idle_gpu_offloadable=True but saves its conditioning still on the borrowed GPU (every other offloadable encoder detaches to CPU), and the consumer discards the result of redux_cond_data.to(device=..., dtype=...) — so with offload enabled the tensor never reaches the session GPU and torch.cat fails with a device mismatch. Planned fix: assign the .to() result, and save the conditioning on CPU.
3. invokeai/backend/util/devices.py:201 / config_default.py — generation_devices: auto expands to every visible CUDA device without consulting a pinned legacy device: cuda:N; an existing install that deliberately excluded a display GPU starts generating on it after upgrade. Planned fix: auto resolves to [device] when device is explicitly set; explicit generation_devices lists still override.
4. invokeai/app/services/session_queue/session_queue_sqlite.py:549 — cancel_all_except_current/delete_all_except_current protect only ONE in-progress item's workflow-call chain (get_current() is LIMIT 1); with N workers, the other workers' waiting parents and pending children get canceled/deleted mid-run. Planned fix: union the chains of all in-progress items.
5. invokeai/app/services/session_processor/session_processor_default.py:636 — the post-dequeue cancel guard can skip a freshly-claimed, non-terminal item when a delayed cancel/clear handler sets the worker's cancel_event against a stale target; the item is left in_progress forever (until restart). Planned fix: cancel the claimed item inside the guard (idempotent).
6. invokeai/app/services/session_queue/session_queue_sqlite.py:756 — a bulk cancel racing clear() hits an unhandled SessionQueueItemNotFoundError between the id-SELECT and the per-item status write → API 500. Pre-existing on main in narrower form; the multi-item loop widens the window. Planned fix: catch-and-skip missing rows.
7. invokeai/app/services/object_serializer/object_serializer_forward_cache.py:72 — eviction does self._cache.pop(self._cache_ids.get()) with no default, but delete() leaves stale ids in _cache_ids → KeyError inside save() once the cache wraps. Pre-existing, but this PR touched the function and its new comment overstates what the added lock fixes. Planned fix: pop with default (mirroring the guard image_files_disk.py already has).
8. invokeai/app/services/session_processor/session_processor_default.py:416 — _clone_session_runner falls back to sharing the single template instance for custom SessionRunnerBase implementations (each worker's start() overwrites the stored cancel_event, so cancel targets the wrong session), and clones DefaultSessionRunner subclasses as plain DefaultSessionRunner. In-tree code only uses DefaultSessionRunner, so this is an API-contract trap rather than a live bug. Planned fix: raise for unsupported runner types when more than one device is configured.
9. invokeai/app/services/invocation_stats/invocation_stats_default.py:60 — an idle-GPU-offloaded encoder's loads are recorded into the borrowed cache's stale CacheStats (usually a defunct object from an already-reset session), so the running session's stats summary omits its own encoder loads. Planned fix: repoint the borrowed cache's stats at the session's stats for the borrow duration.
10. invokeai/app/services/events/events_common.py:159 — progress events resolve the device from the thread-local at emit time, so during an offloaded encoder the UI's GPU badge flips to the borrowed device and back, contradicting the queue item's persisted device column. Cosmetic. Planned fix: prefer queue_item.device in the event builder.

(For completeness, three further candidates from the same review were investigated and refuted — the terminal-status guard in the frontend workflow coordinator already covers trailing progress events, choose_torch_device() normalizes CUDA indices before cache construction, and the device-pool lock registry can't be swapped mid-borrow because there is no processor restart path.)

[JPPhoto]

Some issues:

• invokeai/backend/patches/layer_patcher.py:214: FLUX Control LoRA shape expansion installs a new torch.nn.Parameter without holding MODEL_LOAD_LOCK.read_lock(). A concurrent cold model load holds the write lock while accelerate.init_empty_weights() replaces the process-global register_parameter; the LoRA worker can therefore register its expanded parameter on meta, leaving the running model corrupted or failing later with "Cannot copy out of meta tensor". Test: coordinate two threads so one holds MODEL_LOAD_LOCK.write_lock() inside init_empty_weights() while the other applies a shape-expanding FluxControlLoRALayer, then assert the expanded module parameter remains on its original real device and contains the patched values.

• invokeai/app/invocations/flux_redux.py:95 and invokeai/app/invocations/flux_denoise.py:602: FluxReduxInvocation is newly marked idle_gpu_offloadable=True, but it stores redux_conditioning on the borrowed GPU, and the consumer calls redux_cond_data.to(...) without assigning the returned tensor. When Redux runs on an idle GPU and FLUX denoise resumes on the session GPU, the embeddings remain on the borrowed device and downstream concatenation operates on mixed devices. Test: run Redux with the worker pinned to cuda:0 and the offload context borrowing cuda:1, then consume the saved conditioning in FLUX denoise and assert the stored tensor is CPU-backed and the consumed tensor is assigned to cuda:0.

• invokeai/backend/model_manager/load/model_cache/model_cache.py:1054: the global RAM budget cannot evict entries that are retained only by another device cache, and put() admits the new model even when _make_room_internal() exhausts its local stack without obtaining enough global space. The added test_eviction_cannot_free_ram_held_by_another_device explicitly accepts budget.total_in_use() == 2 * S under a 1.4 * S limit; if the other cache remains idle, this is permanent rather than transient, so max_cache_ram_gb no longer acts as a maximum and repeated cross-device loads can drive the process into OOM. Test: place one shared unlocked model in two caches, request a different model from one cache under a budget smaller than both copies, and assert the implementation either coordinates eviction from the other cache or refuses admission without exceeding RamBudget.max_bytes.

• invokeai/app/services/session_queue/session_queue_sqlite.py:555: _get_current_workflow_call_chain_item_ids() protects the chain of only the single row returned by get_current(... LIMIT 1). With multiple GPU workers, several items can be in_progress; cancel_all_except_current() or delete_all_except_current() can cancel or delete the waiting parents and pending children belonging to every active item except the arbitrarily selected one, breaking those workflow-call executions while their workers continue. Test: create two independent workflow-call chains with one in_progress child in each, invoke both "except current" operations, and assert every active worker's complete ancestor/descendant chain remains intact.

• invokeai/app/services/session_processor/session_processor_default.py:636: the post-dequeue guard skips a newly claimed row whenever worker.cancel_event is set, but does not return that row to pending or mark it canceled. A delayed cancellation event for the previous item can arrive after the worker clears the event but before dequeue() replaces worker.queue_item; the handler sees the previous item, sets the event, and causes the fresh non-terminal item to be abandoned permanently in in_progress. Test: delay the previous item's cancellation handler until after cancel_event.clear(), allow the worker to claim a second item, release the handler before the guard, and assert the second item is canceled or processed rather than left in_progress.

• invokeai/app/services/session_queue/session_queue_sqlite.py:747: _cancel_in_progress_matching() selects IDs in one transaction and updates each ID in later transactions without handling SessionQueueItemNotFoundError. A concurrent queue clear or delete between those operations removes the selected row, causing a bulk cancel request to fail with an unhandled exception; iterating over every active GPU item widens the race introduced by this PR. Test: pause the helper after its ID query, clear or delete the matching rows from another connection, resume it, and assert the bulk cancellation completes while safely ignoring vanished items.

• invokeai/backend/util/devices.py:201 and invokeai/app/services/config/config_default.py:217: the new default generation_devices: auto expands to every visible CUDA device without considering an existing explicit legacy device: cuda:N. An upgraded installation that pinned device to avoid a display GPU or reserve GPUs for other workloads will silently start workers on every GPU. Test: configure device: cuda:1, leave the newly introduced generation_devices unset/defaulted, mock three visible GPUs, and assert upgrade-compatible resolution uses only cuda:1 unless the user explicitly configures multiple generation devices. Document the precedence and migration behavior in docs/src/content/docs/configuration/invokeai-yaml.mdx.

• invokeai/app/services/session_processor/session_processor_default.py:416: multi-device startup does not preserve the SessionRunnerBase contract. A DefaultSessionRunner subclass is cloned as a plain DefaultSessionRunner, losing overrides and state, while any other implementation is reused by all workers even though every start() overwrites its stored cancel event. Cancellation and custom execution behavior can consequently target the wrong worker. Test: start two workers first with a DefaultSessionRunner subclass that overrides run(), then with an independent SessionRunnerBase implementation that stores its cancel event, and assert each worker receives a distinct runner of the original behavioral type with its own event.

• invokeai/app/services/invocation_stats/invocation_stats_default.py:60 and invokeai/app/services/session_processor/session_processor_default.py:146: collect_stats() assigns the graph's CacheStats to the native device cache before _maybe_offload_to_idle_gpu() changes the thread-local device. Encoder loads then update the borrowed cache's existing stats object, which may belong to another or already-reset graph, so the active graph omits its offloaded model hits, misses, sizes, and cache activity. Test: give the native and borrowed caches distinct stats objects, run an offloaded encoder, and assert all encoder cache activity is attributed to the active graph without mutating stale statistics from another graph.

• invokeai/app/services/events/events_common.py:159: progress events derive device from the temporary thread-local session device. During an idle-GPU-offloaded encoder node this is the borrowed GPU, even though the queue item's persisted device identifies the GPU executing the session; the UI badge therefore jumps to the borrowed GPU and back during one queue item. Test: build progress events for a queue item assigned to cuda:0 while the thread is temporarily pinned to cuda:1, and assert the event continues to report the queue item's execution device.

• invokeai/frontend/web/src/features/gallery/components/ImageViewer/context.tsx:126: terminal handling removes the finished item's entry from $progressData but clears the shared $progressEvent and $progressImage without checking which item those globals represent. With two active previews, canceling or failing item A can clear item B's latest global preview; after the map falls back to one item, CurrentImagePreview requires the now-null global image and hides B's still-running preview until another image event arrives. Test: publish image progress for items A and B with B as the latest global event, cancel A, and assert B remains visible as the single full-size preview.

• invokeai/frontend/web/src/features/system/components/SettingsModal/SettingsGenerationDevices.tsx:224: inactive and removable device tags are implemented as Tag elements with role="button" and onClick, but no tabIndex or keyboard activation handler. Keyboard-only users cannot focus or activate the new generation-device controls. Test: render active and inactive device tags, navigate using Tab, activate them with Enter and Space, and assert the same add/remove mutations occur as with pointer clicks.

Some of these are related to newly introduced code, while others are pre-existing assumptions made invalid by this branch's concurrency.

Directly introduced by this branch:

• FLUX Redux offload device mismatch.
• Global RAM budget exceeding its configured cap.
• Post-dequeue cancel guard abandoning an in_progress item.
• generation_devices: auto overriding legacy device pinning.
• Incorrect cloning/reuse of custom session runners.
• Offloaded encoder statistics assigned to the wrong cache.
• Progress events reporting the borrowed GPU.
• Multi-session preview state clearing another active preview.
• Keyboard-inaccessible generation-device controls.

Pre-existing code made unsafe or newly reachable by this branch:

• FLUX Control LoRA parameter registration can overlap model construction. The setattr() predates the PR, but concurrent workers and the new global model-load locking contract create the failing overlap.
• "Except current" workflow operations preserve only one active chain. The workflow-chain helper predates the PR, but multiple simultaneous in_progress items are introduced by this branch.
• The queue deletion race existed in narrower form, but the branch's new _cancel_in_progress_matching() multi-item loop directly adds the reported unhandled path and widens its exposure.

[JPPhoto]

A few issues; I would treat 4 as merge blockers:

• Global RAM budget can remain exceeded after peer-lock contention: invokeai/backend/model_manager/load/model_cache/model_cache.py.
• A stale cancellation event can cancel an unrelated newly claimed job: invokeai/app/services/session_processor/session_processor_default.py.
• FLUX.2 Klein retains conditioning tensors on a borrowed GPU: invokeai/app/invocations/flux2_klein_text_encoder.py.
• User-scoped clearing can reject the owner or interrupt another user's generation: invokeai/app/api/routers/session_queue.py and invokeai/app/services/session_processor/session_processor_default.py.

The other 4 are reasonable follow-ups:

• Incorrect fallback when the latest image-preview owner terminates.
• Stale image-preview entries after queue clearing.
• Misleading "Auto (all GPUs)" UI and schema descriptions.
• Duplicate cancellation counts during concurrent bulk requests.

Here's the list:

• invokeai/backend/model_manager/load/model_cache/model_cache.py:1095: Cross-cache eviction is only best-effort because evict_unlocked_for_peer() returns immediately when the peer cache lock is contended. If the peer contains an unlocked model but briefly holds its cache lock, the requesting cache skips eviction and still admits the new model. Releasing the peer lock does not trigger another budget check, so usage can remain above RamBudget.max_bytes indefinitely. Test: place an unlocked shared model in two caches under a 1.4 * S budget, hold the peer cache lock while the first cache inserts a different S-sized model, then release the lock and assert total usage does not remain at 2 * S.

• invokeai/app/services/session_processor/session_processor_default.py:657: The revised post-dequeue guard prevents an item from remaining in_progress, but it does so by canceling the newly claimed item when a delayed cancellation event belongs to the previous item. The added regression test explicitly expects item 42 to be canceled even though no cancellation targeted it. A stale event can therefore discard an unrelated user's queued generation. Test: delay the previous item's cancellation handler until the worker is between clearing its event and replacing worker.queue_item, claim a second item, then assert the second item runs normally and is not canceled.

• invokeai/app/invocations/flux2_klein_text_encoder.py:77: Flux2KleinTextEncoderInvocation is marked idle_gpu_offloadable=True, but _encode_prompt() returns prompt_embeds and pooled_embeds on the borrowed GPU and invoke() saves them without moving them to CPU. The forward serializer cache retains those GPU tensors after the device-pool lock is released, consuming VRAM on a GPU that may immediately begin another session. Test: run the invocation with its thread pinned to a borrowed CUDA device and assert every tensor passed to context.conditioning.save() is detached and CPU-backed.

• invokeai/app/api/routers/session_queue.py:380 and invokeai/app/services/session_processor/session_processor_default.py:538: User-scoped queue clearing still assumes one current item. get_current() returns one arbitrary in_progress row, so Alice can receive a 403 when Bob's row is selected even while Alice also has an active item. If Alice's row is selected, clear() emits a QueueClearedEvent containing Alice's user_id, but _on_queue_cleared() ignores that field and sets the cancel event for every worker in the queue. Bob's session then stops without Bob's row being deleted or marked terminal, leaving it abandoned as in_progress. Test: run Alice and Bob items concurrently, clear as Alice with each possible get_current() selection, and assert the request succeeds, all Alice rows and workers are cleared, and Bob's worker and database row remain active and unchanged.

• invokeai/frontend/web/src/features/gallery/components/ImageViewer/context.tsx:126 and invokeai/frontend/web/src/features/gallery/components/ImageViewer/CurrentImagePreview.tsx:196: The terminal-state fix only handles completion of a non-global item. If B publishes progress, then A publishes the latest global preview, and A terminates, the handler removes A from $progressData and clears the global atoms. B remains in $progressData, but withProgress depends on the now-null global $progressImage, so B's still-running preview disappears. For successful A completion with auto-switch enabled, the stale completed A preview can remain displayed instead of B. Test: publish image progress for B and then A, terminate A, and assert B immediately becomes the full-size live preview for canceled, failed, and completed terminal states.

• invokeai/frontend/web/src/features/gallery/components/ImageViewer/context.tsx:66: The image-viewer progress map listens for invocation progress and per-item status changes but not queue_cleared. A clear operation deletes all but at most one active row without emitting terminal events for each item, so the remaining $progressData entries survive indefinitely and can appear beside previews from later generations. Test: publish previews for two active items, dispatch a queue-cleared event that covers them, and assert the per-item map, global preview atoms, and resolving state are cleared.

• invokeai/backend/util/devices.py:213, invokeai/app/services/config/config_default.py:217, and invokeai/frontend/web/public/locales/en.json:1859: generation_devices: auto now resolves to the legacy pinned device, but the configuration schema, generated settings, API description, and Settings UI still say that Auto uses every GPU. An upgraded installation with device: cuda:1 displays "Auto (all GPUs)" while starting only one worker, and selecting Auto in the UI cannot produce the advertised behavior without separately changing the legacy setting. Test: render the settings control with device: cuda:1, generation_devices: auto, and multiple available GPUs, and assert the displayed behavior matches the single pinned device. Update docs/src/generated/settings.json through its configuration-description source.

• invokeai/app/services/session_queue/session_queue_sqlite.py:783: _cancel_in_progress_matching() claims to count only rows actually moved to canceled, but it appends any row whose returned status is canceled. If two bulk cancellation requests select the same active row before either update completes, the first request cancels it while the second sees the already-terminal row returned by _set_queue_item_status() and also counts it. Both responses can report that they canceled the same item. Test: synchronize two bulk cancellation calls after their item-ID queries, allow both to continue, and assert the item is counted as newly canceled by only one request.

[lstein]

@JPPhoto Thanks for the fast re-review. All four merge blockers are fixed in 68edb02127, each with the regression test you specified. Also on the branch since your review: 3f78434b65, fixing a cold-cache LoRA-patching self-deadlock on MODEL_LOAD_LOCK.

1. RAM budget stuck exceeded after peer-lock contention (model_cache.py)
evict_unlocked_for_peer() now returns None on lock contention (distinct from "nothing evictable"), and the requester records a reconcile request on each skipped peer. The synchronized decorator gained a post-release hook: the moment the busy peer's current operation releases its lock, it sheds unlocked entries until the shared budget is satisfied. The pending flag persists until the budget is actually met, so overshoot held by locked entries also self-heals when their unlock releases the lock. The hook fires only on the outermost frame (the RLock can be held reentrantly), and reconcile takes only the cache's own lock, so the existing lock ordering is unchanged. Test: your two-cache 1.4 × S scenario — the peer's lock is held from another thread during the insert, and the budget reconciles to S after the lock-holding operation completes.

2. Stale cancellation event canceling an unrelated fresh claim (session_processor_default.py)
The guard now treats the DB row status as the authority. Terminal row → skip. cancel_event set with a non-terminal row → provably stale (a genuine cancel writes the row terminal before emitting its event), so the signal is cleared and the item runs. A terminal re-check after the clear closes the clear's own race window, and a shutdown-raced claim is still canceled so it isn't abandoned in_progress. The old test that expected item 42 to be canceled is replaced by four: stale-event-runs-item (your scenario), genuine-cancel skip, cancel-landing-mid-clear, and shutdown.

3. Klein conditioning retained on a borrowed GPU (flux2_klein_text_encoder.py)
invoke() now saves detach().to("cpu") copies of both embedding tensors, matching flux_text_encoder and flux_redux, with a mirror of the existing redux output-device test asserting everything passed to context.conditioning.save() is detached and CPU-backed.

4. User-scoped clear assuming one current item (session_queue.py / session_processor_default.py)
Moved the cancel-running-items responsibility into the service: clear() now runs _cancel_in_progress_matching() over the scoped filter — the same pattern delete_by_destination() already used — so every in-scope running item is canceled via its own status-changed event (which signals the worker running exactly that item) before its row is deleted. The router's arbitrary get_current() check is gone entirely: a non-admin's scope is exactly their own items, so there is nothing to authorize and no way to 403 Alice on Bob's row or cancel Bob's item. _on_queue_cleared() now honors the event's user_id, so a scoped clear cannot stop other users' workers and abandon their rows. Tests cover the Alice/Bob concurrent scenario at both the service layer (Alice's two running items canceled + deleted; Bob's untouched and still in_progress) and the event-handler layer.

Verification: 1,406 tests pass across the routers/services/model-cache suites plus the new regressions; ruff (CI-pinned 0.11.2) check and format clean.

The four follow-ups (preview-owner termination fallback, stale $progressData after clear, the misleading "Auto (all GPUs)" descriptions, duplicate bulk-cancel counts) are acknowledged and queued as a follow-up round — happy to promote any of them into this PR if you'd rather.

🤖 Generated with Claude Code