[Zacgoose]

Summary

Seven changes: background-failure resilience, cheaper large responses, orchestration and module-loading fixes, and on-demand HTTP workers for interactive users.

1. Keep background loops and timers running when an iteration throws (2718534)

An exception escaping a hosted service's ExecuteAsync stopped the whole host (BackgroundServiceExceptionBehavior defaults to StopHost), and one escaping a System.Threading.Timer callback killed the process. A single out-of-memory error during a large request could restart the container even though the request had already failed cleanly.

• SchedulerService: a failed due-timer check is logged and retried on the next interval.
• WorkPump: a failure taking the instance lock at startup is retried every 5 s instead of faulting the pump.
• MemoryTrimService, cache eviction, the background limiter monitor and the job cleanup timer catch and log their own failures.

2. Write JSON response bodies straight from the PowerShell objects (2ed53ef)

ConvertPsObjectToJson copied the whole result graph into dictionaries and lists and read every object through PSObject.Properties, which builds PowerShell's reserved member sets on each row. The new PsJsonWriter writes the JSON directly with Utf8JsonWriter into pooled 1 MB chunks:

• A plain PSCustomObject is read through its own instance members.
• Other shapes still go through PSObject.Properties.
• Anything it doesn't reproduce exactly (e.g. dictionaries with non-string keys) falls back to the old serializer.

Output is identical; tests compare it with the old path across 20 body shapes.

3. Keep response bodies as UTF-8 bytes from serializer to socket (6cea274)

ScriptResult.BodyBytes is now the body everywhere: the dispatcher writes it in one call, the response cache stores it on disk and in memory, and the trigger checks scan and parse it. Body remains as a decoding convenience, so a large body is no longer also held as a UTF-16 string.

Also fixes a cache bug: a binary List* response (e.g. an image) was cached as its empty text, so a cache hit returned 200 with an empty body.

4. Truncate task error text in the finish transaction (099f09a)

A task's LastError goes into the finish transaction, which cannot split a property over 64 KiB. A very long error made the whole finish fail. It is now capped at 8,000 characters.

5. Shared script modules export what their manifest lists (7a86330, fixes #52)

Workers instantiate each app module from one shared parse. That path exported every function (private ones included), no aliases, and reported version 0.0. It now matches a native manifest import:

• FunctionsToExport and AliasesToExport filter the module's exports, and ModuleVersion is reported. Variables are not exported, matching a native import of a .psm1 without Export-ModuleMember.
• A module that calls Export-ModuleMember itself is imported per worker, because the manifest then filters the module's own export call, which New-Module cannot repeat.
• A large FunctionsToExport list is compressed once per process into the shortest prefix patterns, because New-Module -Function matches every function against every name (about 150 ms per worker on an 846-name list).
• Startup parses only the modules for the pools a node runs: HTTP modules first and in parallel, background-only modules after HTTP is ready.

6. On-demand HTTP workers for interactive requests (ee58073, fd14e21)

New Worker:HttpBurstMax (default 0, off), also settable per SkuProfiles entry and through CRAFT_HTTP_BURST_MAX:

• An interactive (non-API-client) request that finds every HTTP worker busy builds one more in the background, up to HttpBurstMax beyond HttpPoolSize, at most two at a time.
• The waiting request takes whichever worker frees first, so it never waits on the build itself.
• No worker is built while the GC heap is above 80% of its limit.
• API clients, and Craft's own cache refreshes, never hold more than HttpPoolSize workers at once, so burst capacity is only used by interactive requests. When burst is off, no cap is applied.
• After HttpBurstIdleSeconds (default 600) without a user waiting, burst workers are disposed one per tick, back down to HttpPoolSize.
• /healthz now reports workers.http and workers.httpAvailable. Pool metrics and the background limiter's HTTP-pressure check use the live worker count.
• The derived thread-pool minimum includes HttpBurstMax.

Results

Serializer benchmark, 20k rows (CRAFT_BENCH=1, ResponseJsonBench):

Shape	Before	After
Mailbox-rule-like rows (65 MB out)	820 ms, 368 MB allocated, 169 MB peak	102 ms, 78 MB allocated, 66 MB peak
User-like rows (46 MB out)	6,065 ms, 1,871 MB allocated, 649 MB peak	481 ms, 151 MB allocated, 50 MB peak

End to end on a container, a 362 MB response (100k rows): peak container memory went from 2,680 MB to 2,220 MB. Wire bytes are unchanged, except br is ~3% smaller on large bodies because the encoder gets the body in one piece.

Startup on a production-layout app (Linux container, time to HTTP ready):

CPUs	Before	After
1	8.7–9.9 s	7.3–8.3 s
2	8.3–8.5 s	5.8–6.5 s

An extra HTTP worker builds in ~125–170 ms with ~2.2 MB of heap, so burst workers come up quickly.

Client impact

• Response bytes, status codes, content types and handler headers (Location, Content-Disposition, custom headers) are unchanged. The internal X-Craft-Endpoint tag is still removed before sending.
• Module exports: functions not listed in a module's FunctionsToExport are no longer callable from outside that module. An app that called such a helper from another module must export it. This is the same rule a native manifest import always applied.
• Burst is off by default. Nothing changes until HttpBurstMax is set.

Testing

• Full suite: 1,093/1,093.
• New tests:
  • cache round-trips raw bytes from memory and from disk, including a binary body
  • a single-write body at 3 sizes for br and gzip is really compressed and decodes back exactly; uncompressed when not requested
  • a shared import exports the same functions, aliases and version as a native manifest import, across 96 combinations of manifest keys and Export-ModuleMember
  • burst: a user gets a worker while API clients hold the pool, API clients stay capped even with a free burst worker, burst stops at HttpBurstMax, idle workers are trimmed back to the pool
  • SKU profile HttpBurstMax applies only when set; a long task error no longer fails the finish transaction
• Old and new builds run side by side against the same data:
  • Encodings and cache: every endpoint with identity, gzip, br and a browser Accept-Encoding, as a cache miss and a hit. Every compressed response was decoded and compared to the original.
  • Typed responses: handlers returning image/jpeg bytes, a 302 with Location, text/plain, application/pdf with Content-Disposition, custom headers, and pre-serialized JSON strings.
  • Outcome: no response was labeled compressed without being compressed, and headers survived cache hits. Bodies matched the old build, except where a handler's unordered hashtable key order changes per process, as it does between restarts of the old build. The old build returned an empty body for a cached image; the new one returns it intact.
• Docker e2e suite (run-e2e.ps1): 102 of 103 pass.
  • New module checks: unlisted helpers stay private, listed aliases are exported, the manifest version is reported.
  • New http-burst section, also runnable with -BurstOnly: a user call is answered in 114 ms while API clients hold both workers, the pool grows past HttpPoolSize, API clients run two at a time with a burst worker free, and the pool trims back when idle.
  • The one failure, orch-status summary-bounded, also fails on an unmodified dev image. It is a timing-sensitive check, not caused by this PR.
• Host survival: a startup instance-lock failure is retried instead of faulting the pump (new test). An out-of-memory error during a large request now fails only that request.