Design multi-root QDN storage with safe drive expansion and migration

[QuickMythril]

Summary

Design safe support for storing permanent QDN content across multiple configured filesystem roots so a node can continue accepting data when one drive reaches its safe capacity.

This preserves Q-Support record 01d9fab36c1cdd0074b32e82, “QDN content to span multiple drives.” The reporter described a 4 TB drive eventually filling and wanted future content to move to secondary, third, or fourth drives. Comments note that Linux LVM is an existing workaround and also request application-level path selection and movement similar to a multi-library application.

LVM, ZFS, Btrfs, RAID, and bind mounts remain valid operator solutions. This issue concerns a cross-platform Core storage contract. It is design-first work, not an immediate implementation promise.

Current Qortium behavior

Core has:

• one permanent QDN dataPath;
• one temporary/build/staging tempDataPath, defaulting beneath dataPath; and
• one node-wide maxStorageCapacity.

Permanent file placement, lookup, hosted-resource scans, cleanup, safe deletion, relay cache, block-list deletion, original-copy markers, and capacity measurement assume one permanent root. Existing content is deterministically grouped beneath signature directories; locally published copies use an in-directory .original marker that must remain with their content.

First implementation direction

Prefer signature-directory affinity: all permanent files belonging to one transaction signature stay on one root. New signatures may be assigned to another eligible root as earlier roots approach a configured high-water mark.

Do not stripe bytes, individual files, or one signature’s chunks across drives in the first implementation. Multiple roots increase usable capacity; they do not provide RAID, erasure coding, or backup guarantees.

The existing single dataPath configuration and layout must remain the compatible default. tempDataPath may remain one separate staging/build root initially.

Required design

Root registry and path safety

• Give each configured root a stable ID, canonical absolute path, Core-owned identity marker, enabled/read-only state, optional quota, and reserved headroom.
• Reject duplicate paths, nested or overlapping roots, symlink aliases/escapes, non-directories, and overlap with database, backup, lists, temp/staging, or other non-QDN Core data.
• Detect a missing mount before creating an empty replacement directory on the underlying system disk.
• Validate ownership and writability without formatting, mounting, unmounting, or broadly changing filesystem permissions.

Placement and lookup

• Introduce one storage-root abstraction rather than adding path loops to individual call sites.
• Use a durable local placement index for fast lookup, but make it rebuildable by scanning root markers and signature directories.
• A missing or stale index entry should trigger a bounded probe and index repair.
• Define explicit placement for signature content, unsigned _misc files, relay cache, metadata, and original-copy markers.
• Preserve hash validation and normal QDN serving regardless of physical root.

Capacity and cleanup

• Measure usable space, configured quota, and reserve per root without double-counting roots on the same filesystem.
• Retain a clear relationship between per-root limits and global maxStorageCapacity.
• Define current stop-fetch and cleanup thresholds per root and globally.
• A full root must not block new writes to another healthy eligible root.
• Cleanup must target the pressured root, preserve .original content, and never delete through a missing or misidentified mount.

Failure behavior

• Mark an absent, read-only, full, identity-mismatched, or I/O-failing root unhealthy and exclude it from new writes while retaining placement records.
• Treat temporary absence as unavailable content, not deletion.
• Continue blockchain/node operation if QDN roots are degraded; fail QDN writes and hosting clearly if no root is eligible.
• Identity-check and reconcile a reappearing root before returning it to service.
• Expose protected diagnostics for root identity, health, writable state, used/effective capacity, reserve, and migration state.

Expansion, migration, and removal

• Adding a root must not automatically move existing multi-terabyte content; it may receive future signatures according to placement policy.
• Removing a configured root with placed data must require an explicit drain operation.
• Migration should be signature-directory based, throttled, pausable, journaled, restart-safe, and idempotent.
• Cross-filesystem moves must copy to destination staging, verify bytes/hashes and durability, atomically finalize where supported, update placement state, and only then delete the source.
• A crash or drive failure at each phase must leave at least one valid copy or a clearly recoverable journal state.
• Define backup, restore, rollback, rebalance, and previous-Core-version behavior.

First deliverable

Produce an ADR/design and implementation map covering:

• configuration schema and single-root compatibility;
• root identity and placement-index persistence/rebuild;
• placement, capacity, cleanup, and failure state machines;
• add, drain, migrate, remove, and optional rebalance workflows;
• path-containment and mount-identity threat model;
• restricted admin API plus Node/Home settings and diagnostics;
• staged implementation and rollback boundaries; and
• automated failure-injection and real multi-drive validation.

Acceptance criteria for eventual implementation

• A default single-root node behaves exactly as before.
• An existing node can add another root without relocating current content.
• New signature directories move to an eligible root when the preferred root reaches its safe limit.
• Content remains readable and hash-valid across restart and root-order changes.
• A missing drive cannot silently become an empty directory on another filesystem and does not trigger destructive cleanup.
• Adding, draining, migrating, and removing a root are resumable and preserve protected/original content.
• Capacity, cleanup, relay, unsigned-content, staging, and storage-policy behavior are correct across roots.
• All-roots-unavailable behavior fails QDN closed without stopping normal blockchain operation.
• Linux and at least one non-Linux host are tested with distinct filesystems.
• Existing LVM/ZFS/Btrfs, bind-mount, and single-root deployments remain supported alternatives.