Add Diskcache and Redis shared-storage backends

LocalSharedStorage only reaches one container, which fragments across pods
behind a load balancer (Dash Enterprise scales apps multi-pod). Add two more
BaseSharedStorage backends, selected via shared_storage=:

- DiskcacheSharedStorage: KV + append-log pub/sub on a diskcache.Cache, shared
  by every process on one host. Single-machine parity; not for pods.
- RedisSharedStorage: KV + Redis Streams pub/sub (atomic INCR+XADD, MAXLEN-
  bounded, SharedStorageGap past the trimmed floor). Correct across pods.

Both reuse a shared PollingSubscription. Tests mirror the local suite; the
Redis tests run against the CI redis service and skip when none is reachable.

Author: T4rk1n

.ai/ARCHITECTURE.md

@@ -877,47 +877,78 @@ gap rather than a silent hole). A `Subscription` is iterable synchronously
 (`for`) or asynchronously (`async for`), is a context manager, and has
 `close()`.
 
-### Default Backend: `LocalSharedStorage`
+### Backends
 
-In-memory, owner-elected — no external service required:
+Three backends ship, differing only in **how far state reaches**. Pick by the
+deployment topology — critically, whether the app runs behind a load balancer as
+more than one container/pod (see [Deployment Topology](#deployment-topology-and-shared-storage) below).
 
-- On first use, every worker races to become the single **owner** for the
-  machine by binding a stable address (`AF_UNIX` socket on POSIX, TCP loopback
-  on Windows). The winner hosts the authoritative in-memory engine; the losers
-  become clients that proxy over the socket.
-- The bind *is* the lease: when the owner dies the address frees and a surviving
-  worker re-elects — **coming up cold** (state is not durable; a re-election
-  starts with an empty store).
-- A single-process deployment is its own owner and pays **no socket overhead**.
-- The channel is gated by a per-owner random token written to a `0600`
-  advertisement file, so only same-host clients that can read it may attach.
+| Backend | Backing store | Reaches | Extra |
+|---------|---------------|---------|-------|
+| `LocalSharedStorage` *(default)* | in-memory, owner-elected socket | one **container** (all its workers) | none |
+| `DiskcacheSharedStorage` | a `diskcache.Cache` | one **host** (all processes sharing the dir) | `dash[diskcache]` |
+| `RedisSharedStorage` | Redis (Streams for pub/sub) | **any** process/container/pod | `dash[celery]` (redis) |
 
-Constructor knobs (rarely needed):
+All three implement the same `BaseSharedStorage` contract (KV + ordered,
+replayable pub/sub with `SharedStorageGap` on buffer overrun), so app code is
+identical across them — only the constructor differs.
 
 ```python
-from dash._shared_storage import LocalSharedStorage
-
-Dash(shared_storage=LocalSharedStorage(
-    namespace=None,     # defaults to a hash of cwd + argv[0] (isolates apps)
-    buffer_size=2048,   # per-topic replay buffer depth
-))
-```
-
-### Configuration
+from dash import Dash
+from dash._shared_storage import (
+    LocalSharedStorage, DiskcacheSharedStorage, RedisSharedStorage,
+)
 
-```python
-Dash(shared_storage=LocalSharedStorage)   # default
-Dash(shared_storage=None)                 # disabled
-Dash(shared_storage=MyBackend())          # custom backend
+Dash(shared_storage=LocalSharedStorage)                       # default
+Dash(shared_storage=DiskcacheSharedStorage(directory="/tmp/ss"))
+Dash(shared_storage=RedisSharedStorage(url="redis://host:6379"))
+Dash(shared_storage=None)                                     # disabled
 ```
 
 When disabled, accessing `app.shared_storage` / `dash.ctx.shared_storage`
 raises `SharedStorageError`.
 
+**`LocalSharedStorage`** — in-memory, no external service. On first use every
+worker races to become the single **owner** for the machine by binding a stable
+address (`AF_UNIX` on POSIX, `127.0.0.1` loopback on Windows); the winner hosts
+the engine, losers proxy over the socket. The bind *is* the lease — when the
+owner dies the address frees and a survivor re-elects **cold** (state is not
+durable). A single-process deployment is its own owner and pays no socket
+overhead. Knobs: `namespace` (defaults to a hash of cwd + argv[0]),
+`buffer_size` (default 2048).
+
+**`DiskcacheSharedStorage`** — KV + an append-log pub/sub on a `diskcache.Cache`,
+the same store `DiskcacheManager` uses; pass an existing `cache` to share one.
+Every process on one host that opens the same directory shares state, so it
+gives multi-worker parity with the local backend without a socket. **Not for
+pods** — each pod has its own ephemeral disk.
+
+**`RedisSharedStorage`** — KV on Redis strings, pub/sub on a Redis Stream per
+topic (an atomic `INCR`+`XADD` script keeps sequences ordered; `MAXLEN` bounds
+the replay window; a subscriber past the trimmed floor gets `SharedStorageGap`).
+Redis is the single source of truth, so no owner election. Pass a `url`
+(defaults to `$REDIS_URL`) or an existing `client` to reuse a connection pool.
+This is the only backend correct for **horizontally-scaled, multi-pod**
+deployments.
+
+### Deployment Topology and Shared Storage
+
+The default `LocalSharedStorage` is **per-container**: its socket/loopback
+election only reaches processes in the same network + filesystem namespace.
+
+- **Single process / single pod** (e.g. Plotly Cloud apps): fine — one owner,
+  nothing to fragment.
+- **Multiple gunicorn workers in one container**: fine — they share the pod's
+  loopback/socket and elect one owner.
+- **Multiple pods behind a load balancer** (e.g. Dash Enterprise apps scaled by
+  an HPA, routed round-robin with no session affinity): **each pod elects its own
+  isolated owner**, so `set()`/`publish()` on one pod are invisible on another.
+  This works at 1 pod and fragments silently once it scales — use
+  `RedisSharedStorage` (one Redis shared by all pods) instead.
+
 **Custom backends** subclass `BaseSharedStorage` and implement `get` / `set` /
 `delete` / `publish` / `subscribe` (plus optional `start` / `close`, called once
-per worker). This is the seam for a Redis- or database-backed store that shares
-state across *machines*, not just processes on one host.
+per worker) — the seam for e.g. a database-backed store.
 
 ### Module Layout
 
@@ -926,12 +957,16 @@ state across *machines*, not just processes on one host.
 | `_shared_storage/base.py` | Abstract interface: `BaseSharedStorage`, `Subscription`, `SharedStorageError`, `SharedStorageGap` |
 | `_shared_storage/_engine.py` | `StoreEngine`: authoritative in-memory kv map + per-topic ordered log with bounded replay buffer (thread-safe, transport-agnostic) |
 | `_shared_storage/local.py` | `LocalSharedStorage`: owner election; owner hosts the engine, clients proxy |
-| `_shared_storage/_transport.py` | Length-prefixed, token-gated socket transport |
+| `_shared_storage/diskcache.py` | `DiskcacheSharedStorage`: KV + append-log pub/sub on a `diskcache.Cache` |
+| `_shared_storage/redis.py` | `RedisSharedStorage`: KV + Redis Streams pub/sub (atomic `INCR`+`XADD`) |
+| `_shared_storage/_polling.py` | `PollingSubscription`: shared poll-loop subscription for the diskcache/Redis backends |
+| `_shared_storage/_transport.py` | Length-prefixed, token-gated socket transport (local backend) |
 | `_shared_storage/_codec.py` | msgspec/json codec (data-only) |
 | `dash.py` | `shared_storage` constructor arg + lazy `app.shared_storage` property |
 | `_callback_context.py` | `dash.ctx.shared_storage` accessor |
 
-Requires the `msgspec` dependency (declared in `requirements/install.txt`).
+Requires `msgspec` (in `requirements/install.txt`); `DiskcacheSharedStorage`
+needs the `diskcache` extra and `RedisSharedStorage` the `celery` extra (redis).
 
 ## Async Callbacks
 

.github/workflows/testing.yml

@@ -72,6 +72,21 @@ jobs:
       matrix:
         python-version: ["3.9", "3.12"]
 
+    # Redis service for the RedisSharedStorage backend tests.
+    services:
+      redis:
+        image: redis:6
+        ports:
+          - 6379:6379
+        options: >-
+          --health-cmd "redis-cli ping"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+
+    env:
+      REDIS_URL: redis://localhost:6379
+
     steps:
       - name: Checkout repository
         uses: actions/checkout@v4
@@ -108,7 +123,7 @@ jobs:
         run: |
           python -m pip install --upgrade pip wheel
           python -m pip install "setuptools<80.0.0"
-          find packages -name dash-*.whl -print -exec sh -c 'pip install "{}[dev,ci,testing,diskcache]"' \;
+          find packages -name dash-*.whl -print -exec sh -c 'pip install "{}[dev,ci,testing,diskcache,celery]"' \;
 
       - name: Install dash-renderer dependencies
         working-directory: dash/dash-renderer

CHANGELOG.md

@@ -5,7 +5,7 @@ This project adheres to [Semantic Versioning](https://semver.org/).
 ## [Unreleased]
 
 ### Added
-- [#3930](https://github.com/plotly/dash/pull/3930) Shared storage: a backend-agnostic state manager available on every app via `dash.ctx.shared_storage` (and `app.shared_storage`). It offers a cross-process key/value store (`get`/`set`/`delete`) and ordered publish/subscribe (`publish`/`subscribe`) for sharing state between callbacks or across worker processes without an external service. The default `LocalSharedStorage` backend elects a single owner process per machine (binding an `AF_UNIX` socket on POSIX, TCP loopback on Windows) and serves the rest; a single-process deployment is its own owner and pays no socket overhead. Subscriptions are ordered and replayable — a consumer that reconnects resumes from its last-seen message from a bounded buffer, and a buffer overrun surfaces as an explicit gap rather than a silent loss. Values must be JSON-compatible (like `dcc.Store`). Started lazily on first use, so it costs nothing until touched; pass `shared_storage=None` to `Dash(...)` to disable, or a `BaseSharedStorage` subclass/instance to swap the backend. The wire codec is msgspec (never pickle).
+- [#3930](https://github.com/plotly/dash/pull/3930) Shared storage: a backend-agnostic state manager available on every app via `dash.ctx.shared_storage` (and `app.shared_storage`). It offers a cross-process key/value store (`get`/`set`/`delete`) and ordered, replayable publish/subscribe (`publish`/`subscribe`) for sharing state between callbacks or across worker processes without an external service. Subscriptions resume from a caller's last-seen sequence out of a bounded buffer, and a buffer overrun surfaces as an explicit gap rather than a silent loss. Values must be JSON-compatible (like `dcc.Store`). Started lazily on first use, so it costs nothing until touched; pass `shared_storage=None` to `Dash(...)` to disable. Three backends ship, selected by passing a `BaseSharedStorage` instance to `shared_storage=`: the default `LocalSharedStorage` (in-memory, elects one owner process per machine over an `AF_UNIX`/loopback socket — correct within a single container, including across multiple gunicorn workers); `DiskcacheSharedStorage` (on a `diskcache.Cache`, shared by every process on one host — parity for single-machine deployments, not for multi-pod ones with ephemeral disks); and `RedisSharedStorage` (on Redis, using Redis Streams for the ordered pub/sub — the backend for horizontally-scaled deployments behind a load balancer, e.g. apps scaled across pods).
 - [#3646](https://github.com/plotly/dash/pull/3646) Experimental support for React 19. The default is still React 18.3.1; to use React 19 set the environment variable `REACT_VERSION=19.2.4` before running your app, or call `dash._dash_renderer._set_react_version("19.2.4")` inside the app. React 19 has no official UMD builds, so Dash serves the [`umd-react`](https://www.npmjs.com/package/umd-react) package, together with a compatibility shim loaded after react-dom and before any component package. The shim keeps component libraries built against React <=18 (e.g. dash-bootstrap-components, dash-mantine-components) working under React 19: it stubs the removed `ReactCurrentOwner` internals, redirects the legacy element `$$typeof` symbol so pre-bundled React 18 jsx-runtimes produce elements React 19 accepts (error #525), and exposes a global `react/jsx-runtime` (`window.ReactJSXRuntime`) that Dash's own component bundles externalize to. Component library authors adopting this convention should copy the defensive `jsxRuntimeExternal` webpack external from `components/dash-core-components/webpack.config.js` rather than a bare `'ReactJSXRuntime'` string: it falls back to a `React.createElement`-based runtime when the global is missing, so the same build also works on Dash versions older than this release.
 
 ### Removed

dash/__init__.py

@@ -45,7 +45,9 @@
 
 from ._shared_storage import (  # noqa: F401,E402
     BaseSharedStorage,
+    DiskcacheSharedStorage,
     LocalSharedStorage,
+    RedisSharedStorage,
     SharedStorageError,
     SharedStorageGap,
     Subscription,
@@ -103,7 +105,9 @@ def _jupyter_nbextension_paths():
     "hooks",
     "stringify_id",
     "BaseSharedStorage",
+    "DiskcacheSharedStorage",
     "LocalSharedStorage",
+    "RedisSharedStorage",
     "SharedStorageError",
     "SharedStorageGap",
     "Subscription",

dash/_shared_storage/__init__.py

@@ -1,7 +1,10 @@
 """Backend-agnostic shared state + pub/sub (state manager).
 
-Exposes the ``BaseSharedStorage`` interface and the default
-``LocalSharedStorage`` backend (owner-elected, cross-process).
+Exposes the ``BaseSharedStorage`` interface and its backends:
+
+- ``LocalSharedStorage`` (default): in-memory, owner-elected -- one container.
+- ``DiskcacheSharedStorage``: on a ``diskcache.Cache`` -- one machine, not pods.
+- ``RedisSharedStorage``: on Redis -- across processes and containers.
 """
 
 from .base import (
@@ -10,11 +13,15 @@
     SharedStorageGap,
     Subscription,
 )
+from .diskcache import DiskcacheSharedStorage
 from .local import LocalSharedStorage
+from .redis import RedisSharedStorage
 
 __all__ = [
     "BaseSharedStorage",
+    "DiskcacheSharedStorage",
     "LocalSharedStorage",
+    "RedisSharedStorage",
     "SharedStorageError",
     "SharedStorageGap",
     "Subscription",

dash/_shared_storage/_polling.py

@@ -0,0 +1,73 @@
+"""Subscription for poll-based backends.
+
+A backend that can answer ``poll(topic, after_seq, timeout) -> PollResult``
+builds its live subscriptions from this: it drives the poll loop, tracks the
+cursor, resumes from it on every call, and turns a buffer overrun into
+``SharedStorageGap``. ``LocalSharedStorage`` has its own reconnecting
+subscription (it also handles owner re-election); this serves the diskcache and
+Redis backends, whose store is always reachable.
+"""
+import asyncio
+import threading
+from typing import Callable
+
+from ._engine import PollResult
+from .base import SharedStorageGap, Subscription
+
+PollFn = Callable[[str, int, float], PollResult]
+
+
+class PollingSubscription(Subscription):
+    """Iterate a topic by repeatedly polling ``poll_fn`` from a moving cursor."""
+
+    def __init__(
+        self, topic: str, start_seq: int, poll_fn: PollFn, poll_timeout: float
+    ):
+        self._topic = topic
+        self._cursor = start_seq
+        self._poll_fn = poll_fn
+        self._poll_timeout = poll_timeout
+        self._closed = threading.Event()
+
+    def close(self) -> None:
+        self._closed.set()
+
+    def _gap(self) -> SharedStorageGap:
+        return SharedStorageGap(f"replay buffer overran on topic {self._topic!r}")
+
+    def __iter__(self):
+        try:
+            while not self._closed.is_set():
+                res = self._poll_fn(self._topic, self._cursor, self._poll_timeout)
+                if res.gap:
+                    raise self._gap()
+                yield from res.messages
+                self._cursor = res.last_seq
+        finally:
+            self.close()
+
+    def __aiter__(self):
+        return self._aiter()
+
+    async def _aiter(self):
+        loop = asyncio.get_running_loop()
+        try:
+            while not self._closed.is_set():
+                try:
+                    res = await loop.run_in_executor(
+                        None,
+                        self._poll_fn,
+                        self._topic,
+                        self._cursor,
+                        self._poll_timeout,
+                    )
+                except RuntimeError:
+                    # The loop/executor is shutting down -- end cleanly.
+                    break
+                if res.gap:
+                    raise self._gap()
+                for message in res.messages:
+                    yield message
+                self._cursor = res.last_seq
+        finally:
+            self.close()