Prove namespace noisy-neighbor containment for shared Server deployments

[rmcdaniel]

Problem

Namespace authorization prevents cross-tenant data access, but a shared Durable Workflow Server also needs resource isolation. One namespace must not monopolize API capacity, task leasing, scheduler work, database growth, Redis state, or downstream service-call budgets until unrelated namespaces stop making progress.

Server already has queue/namespace lease and dispatch admission controls plus task-queue fairness primitives. This work must audit those controls as one system, close uncovered paths, and prove an adversarial namespace remains contained under realistic constrained capacity.

Required contract

• Enforce server-owned per-namespace admission independently of caller-supplied queue, priority, or fairness metadata.
• Preserve bounded progress for quiet namespaces while another namespace floods workflow starts, activities, timers, signals, queries, updates, schedules, child workflows, and Nexus/service calls.
• Bound namespace-owned active leases, pending tasks, dispatch rate, API concurrency/rate, open workflows, history growth, timers/schedules, external payload usage, and other durable/cardinality growth that can exhaust shared infrastructure.
• Return explicit retryable admission responses and operator-visible metrics identifying the namespace and exhausted budget; do not silently drop durable work.
• Keep counters and leases correct through process restart, Redis interruption, expired leases, and retry/replay.
• Define which limits are configurable defaults, per-namespace overrides, and hard server ceilings. Unsafe unlimited defaults must be explicit rather than accidental.

Adversarial evidence

Build a repeatable constrained-cell experiment with at least one noisy namespace and one control namespace. It must cover:

• sustained workflow-start and activity-dispatch saturation;
• same-queue and different-queue contention;
• timer, signal, query, update, schedule, child-workflow, and Nexus/service-call storms;
• large/replay-heavy histories and external payload pressure;
• worker/server restart and Redis interruption during saturation;
• database, Redis, memory, CPU, queue depth, rejection counts, and p50/p95 control-namespace latency.

The control namespace must continue starting and completing a documented minimum workload within a bounded latency envelope while the noisy namespace receives deterministic throttling. After pressure stops, both namespaces must recover without manual data repair.

Completion

• Inventory every shared resource and admission path, with existing coverage and gaps.
• Implement missing namespace-level limits/fair scheduling and tests.
• Publish a reusable experiment in this repository, not provider-specific orchestration.
• Record an exact Server/image tuple and adversarial results on this issue.
• State the proven shared-server operating envelope and remaining non-isolated resources.
• Only then recommend whether mutually untrusted namespaces can safely share one small Server deployment.

This is a Server product capability. Hosting products can consume the result, but provider topology and commercial plan decisions are out of scope here.

[rmcdaniel]

Implementation is visible in draft PR #92. The first landed-ready slice makes configured workflow/activity lease admission fail closed when its shared cache/lock authority is unavailable, while preserving the backward-compatible unlimited path. Focused verification: 12 tests, 125 assertions.

This is not completion evidence for shared tenancy. Remaining work is the API and durable-cardinality budget inventory/implementation, cross-process Nexus limits, operator rejection metrics, and the constrained noisy/control namespace experiment required by this issue.

[rmcdaniel]

First implementation slice merged in #92 (b815aa90): configured workflow/activity lease admission now fails closed if shared cache/lock authority is unavailable, while the explicit unlimited default remains backward compatible. Work continues in separate reviewable slices; this issue remains open and does not yet claim shared-tenancy safety.

[rmcdaniel]

Second bounded implementation slice landed in #93 at b400112a35b1ba9948954e836babcd3d6ca78c06.

This adds per-namespace request-rate and concurrent-request admission across customer control-plane routes, including workflows, schedules, management APIs, and Nexus/service execution. Limits support defaults, namespace overrides, non-bypassable hard ceilings, retryable 429/503 responses, fail-closed cache/lock behavior, and bounded per-namespace rejection metrics. Admin diagnostics/remediation remain reachable while a namespace is exhausted.

GitHub qualification passed: 1,940 feature/contract tests with 35,739 assertions, MySQL replay/query topology, bounded-growth contract, and the polling-cache perf smoke.

This still is not shared-tenancy proof. Remaining work includes worker-plane containment, external-payload byte budgets, durable-state/cardinality limits and scheduler fairness, followed by the required adversarial noisy/control namespace experiment and published operating envelope.

[rmcdaniel]

Worker-plane long-poll fairness landed in #94 at 2e0bf0b3113fe46f4be7b93315f9b8e9322caca3.

The Server now supports optional per-namespace caps for held workflow/activity and query/update-validation worker polls. Namespace and global slots are acquired as one reservation, invalid or unavailable shared admission authority fails closed when isolation is configured, and workflow, activity, query, update-validation, and Nexus worker paths carry namespace scope.

GitHub qualification passed: 1,941 feature/contract tests with 35,746 assertions, MySQL replay/query topology, bounded-growth contract, and the polling-cache performance smoke.

This remains partial containment. Cumulative external-payload quotas, durable-state/cardinality limits, scheduler fairness, and the constrained noisy/control namespace experiment remain before any shared-tenancy recommendation.

[rmcdaniel]

Cumulative external-payload containment landed in #95 at 35efdc1a349c15ca235530c0dd0c069912fb3791.

The Server now supports optional default/per-namespace cumulative byte and object limits, non-bypassable hard ceilings, database-serialized concurrent admission, fail-closed policy handling, idempotent duplicate registration, typed retryable 429/503 outcomes, and namespace-scoped usage/rejection metrics. The public OpenAPI and transport manifest describe the same outcomes.

GitHub qualification passed: 1,867 feature tests with 34,517 assertions, MySQL replay/query topology, repository and bounded-growth contracts, and the polling-cache performance smoke.

This remains partial containment. Durable workflow/history/timer/schedule cardinality limits, scheduler fairness, and the constrained noisy/control namespace experiment remain before any shared-tenancy recommendation.

[rmcdaniel]

Durable-state cardinality containment landed in #96 at 99c853183fcf0ed1ffaa945592db0bd24750edf7.

The Server now supports database-authoritative default, per-namespace, and hard-ceiling limits for workflow instances, workflow runs, open runs, schedules, schedule-history events, and worker registrations. Direct starts, standalone activities, child workflows, continue-as-new, schedules, and worker registration are covered transactionally; duplicate/idempotent operations that do not grow state remain usable at capacity. Rejections are typed and observable.

GitHub qualification passed: 1,961 feature/corpus tests, MySQL replay/query topology, repository and bounded-growth contracts, and the 1,000-run polling-cache smoke.

This remains partial containment. History/task/timer/stream growth, scheduler fairness, cross-process service-call limits, and the adversarial noisy/control experiment still remain before a shared-tenancy recommendation.

[rmcdaniel]

Cross-process service-call containment landed in #97 at 72627fb0aa3bff6b9c49c61ca97443c1fa152ab7.

Configured operation and aggregate caller-namespace rate/concurrency budgets now use shared polling-cache authority across Server processes, with default and per-namespace limits, non-bypassable hard ceilings, typed retryable 429/503 outcomes, bounded rejection metrics, replay-safe authorization, and finite-TTL recovery for abandoned async reservations. The explicit unlimited default avoids cache dependence.

GitHub qualification passed: the full feature suite, MySQL replay/query topology, repository and bounded-growth contracts, and the 1,000-run polling-cache smoke.

This remains partial containment. History/task/timer/stream growth, scheduler fairness, and the adversarial noisy/control namespace experiment remain before a shared-tenancy recommendation.