From 760d4309e65d2fcb39c234f0349ea8002192a649 Mon Sep 17 00:00:00 2001 From: Tim Conley Date: Wed, 9 Sep 2026 10:29:53 -0700 Subject: [PATCH 1/4] docs: add System Nexus design --- docs/system-nexus.md | 74 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 docs/system-nexus.md diff --git a/docs/system-nexus.md b/docs/system-nexus.md new file mode 100644 index 00000000..aff4cf10 --- /dev/null +++ b/docs/system-nexus.md @@ -0,0 +1,74 @@ +# System Nexus + +System Nexus operations are ordinary Nexus operations that target the reserved +`__temporal_system` endpoint. The Temporal server handles them directly rather +than routing them to a user-defined Nexus handler. + +## Serialization + +The System Nexus operation input (the *System Nexus envelope*) must remain +readable by the server. SDKs therefore cannot apply their user data converter +to the envelope as a whole: its outer representation must reach the server as +binary protobuf. The encoded envelope is marked with +`__temporal_system_payload` metadata so SDK infrastructure can recognize it in +later processing. + +### Inner payloads + +Payloads contained within the envelope—for example, Signal-with-Start input +arguments, memo, headers, and search attributes—must use the application's +data converter so the target Workflow can consume them. + +When an SDK processes payloads, it recognizes a System Nexus envelope by its +payload marker. Instead of applying its operation to the outer envelope, it +deserializes the envelope, applies generic payload visitation to all nested +application payloads, and serializes the envelope again. This applies to +payload codecs and external storage today. + +### Serialization context + +Each System Nexus operation can require a different serialization context, and +that context can differ from an ordinary Nexus operation's context. For +example, Signal-with-Start's nested payloads require the target Workflow's +serialization context so the target can decode them. + +SDKs derive the context from the System Nexus operation through generated code. +The current model is an operation annotation naming a function that receives +the operation input and returns a serialization context. + +## Interception + +Each System Nexus operation has two interception points with distinct purposes. + +The operation-specific interception point receives the complete generated +request type. It exposes the arguments, headers, and other values delivered to +the target and is the appropriate point for tracing propagation and most +operation-specific interception behavior. + +The generic System Nexus interception point receives the outer Nexus-operation +scheduling input. It is functionally equivalent to the ordinary generic Nexus +interception point but permits policies that apply to all System Nexus +operations. For example, an authentication proxy may require headers that do +not belong on the server-facing System Nexus request. + +## Language-specific considerations + +### TypeScript + +In TypeScript, creating the protobuf-binary envelope inside the Workflow +isolate is difficult because protobufjs requires writes. The isolate therefore +serializes the envelope with the normal JSON converter. Worker-side payload +processing converts it to protobuf binary before it reaches the server. This +conversion happens regardless of whether a payload codec or external storage +is configured. + +The serialization-context factory is defined for the user model type, while +the transfer type is what remains after leaving the isolate. TypeScript +therefore stores the derived context in `__temporal_system_context` metadata on +the JSON envelope so it can be reused outside the isolate. + +## Adding an operation + +Adding a System Nexus operation requires generated transfer types, a +serialization-context factory, a discoverable protobuf service and method +descriptor, and payload-visitor coverage for every nested payload field. From 190434d0e444db9aff0448028b5d8982d824a3ef Mon Sep 17 00:00:00 2001 From: Tim Conley Date: Wed, 9 Sep 2026 10:30:12 -0700 Subject: [PATCH 2/4] docs: reorder System Nexus design sections --- docs/system-nexus.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/system-nexus.md b/docs/system-nexus.md index aff4cf10..a0afeb5a 100644 --- a/docs/system-nexus.md +++ b/docs/system-nexus.md @@ -51,6 +51,12 @@ interception point but permits policies that apply to all System Nexus operations. For example, an authentication proxy may require headers that do not belong on the server-facing System Nexus request. +## Adding an operation + +Adding a System Nexus operation requires generated transfer types, a +serialization-context factory, a discoverable protobuf service and method +descriptor, and payload-visitor coverage for every nested payload field. + ## Language-specific considerations ### TypeScript @@ -66,9 +72,3 @@ The serialization-context factory is defined for the user model type, while the transfer type is what remains after leaving the isolate. TypeScript therefore stores the derived context in `__temporal_system_context` metadata on the JSON envelope so it can be reused outside the isolate. - -## Adding an operation - -Adding a System Nexus operation requires generated transfer types, a -serialization-context factory, a discoverable protobuf service and method -descriptor, and payload-visitor coverage for every nested payload field. From 83fbba21e92c619a5dbadd89e1eeba77d9504c9c Mon Sep 17 00:00:00 2001 From: Tim Conley Date: Wed, 9 Sep 2026 10:33:12 -0700 Subject: [PATCH 3/4] docs: move System Nexus design into feature --- docs/system-nexus.md => features/system_nexus/README.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/system-nexus.md => features/system_nexus/README.md (100%) diff --git a/docs/system-nexus.md b/features/system_nexus/README.md similarity index 100% rename from docs/system-nexus.md rename to features/system_nexus/README.md From c0694f42a1b227c327ff06776c927c5ebb2864e3 Mon Sep 17 00:00:00 2001 From: Tim Conley Date: Wed, 9 Sep 2026 10:34:45 -0700 Subject: [PATCH 4/4] docs: restore System Nexus design prose --- features/system_nexus/README.md | 63 ++++++++------------------------- 1 file changed, 14 insertions(+), 49 deletions(-) diff --git a/features/system_nexus/README.md b/features/system_nexus/README.md index a0afeb5a..2e2b24a7 100644 --- a/features/system_nexus/README.md +++ b/features/system_nexus/README.md @@ -1,55 +1,28 @@ # System Nexus -System Nexus operations are ordinary Nexus operations that target the reserved -`__temporal_system` endpoint. The Temporal server handles them directly rather -than routing them to a user-defined Nexus handler. +System Nexus operations are normal nexus operations which happen to target the `__temporal_system` endpoint. They are handled by the server rather than a user's own defined nexus handler. ## Serialization -The System Nexus operation input (the *System Nexus envelope*) must remain -readable by the server. SDKs therefore cannot apply their user data converter -to the envelope as a whole: its outer representation must reach the server as -binary protobuf. The encoded envelope is marked with -`__temporal_system_payload` metadata so SDK infrastructure can recognize it in -later processing. +Because the operations need to be read by the server, the nexus operation's input, which we'll call the "system nexus envelope" needs to be handled uniquely. It can't be serialized with the user's data converter (payload converter, codec, or external storage) because the server would then be unable to read it. Instead, it needs to reach the server specifically encoded with binary protobuf. It should additionally have `__temporal_system_payload` metadata for use in later recognition. -### Inner payloads +### Inner Payloads -Payloads contained within the envelope—for example, Signal-with-Start input -arguments, memo, headers, and search attributes—must use the application's -data converter so the target Workflow can consume them. +The payloads which are contained within the envelope however, like the input arguments, memos, etc in SignalWithStart, do need to use the user's dataconverter so that they are usable when they reach the target workflow. To accomplish this, anything that processes payloads needs to be updated to handle the recognition and handling of system nexus envelopes. This includes codec application and external storage today. Upon reaching a system nexus envelope, the code (usually a generic visitor for code sharing) should recognize a system nexus envelope by its payload marker metadata. Instead of applying the current operation to it, it deserializes that payload, and uses generic payload visitation on the resulting protobuf object to apply the operation to all contained payloads instead. Then it reserializes the envelope. -When an SDK processes payloads, it recognizes a System Nexus envelope by its -payload marker. Instead of applying its operation to the outer envelope, it -deserializes the envelope, applies generic payload visitation to all nested -application payloads, and serializes the envelope again. This applies to -payload codecs and external storage today. +### Context -### Serialization context +Each system nexus operation could potentially need a different serialization context from each other, and often not that used for a normal nexus operation. For example, signal with start's internal user payloads must have the target workflow serialization context, so that they can be appropriately deserialized/decoded by the target workflow. For this reason, serialization context for system nexus operations must be derived from the operations themselves through generated code. Currently this is a function annotation which is given the operation input and must produce a serialization context. -Each System Nexus operation can require a different serialization context, and -that context can differ from an ordinary Nexus operation's context. For -example, Signal-with-Start's nested payloads require the target Workflow's -serialization context so the target can decode them. +### Responses -SDKs derive the context from the System Nexus operation through generated code. -The current model is an operation annotation naming a function that receives -the operation input and returns a serialization context. +Responses to the system nexus operation will also be marked as system payloads. They will go through the same process and should use the same context as that used for its outgoing request. ## Interception -Each System Nexus operation has two interception points with distinct purposes. +Each system nexus operation needs two points of interception which serve different purposes. First, each operation has its own interception point specific to that operation, which receives the whole input request type. This gives access to the arguments/headers/etc which are received by the target, and should be used to implement tracing headers and most other interception behavior. -The operation-specific interception point receives the complete generated -request type. It exposes the arguments, headers, and other values delivered to -the target and is the appropriate point for tracing propagation and most -operation-specific interception behavior. - -The generic System Nexus interception point receives the outer Nexus-operation -scheduling input. It is functionally equivalent to the ordinary generic Nexus -interception point but permits policies that apply to all System Nexus -operations. For example, an authentication proxy may require headers that do -not belong on the server-facing System Nexus request. +The second interception point is a generic one for all system nexus operations. This receives the actual outer nexus operation's input rather than the envelope. It is functionally equivalent to the existing nexus interception point, but distinguished because many interceptors may choose to do different things (tracing ones should usually log in the specific one rather than the generic one for instance). This is primarily important for a scenario such as needing to apply headers for an auth proxy. Any headers attached here would otherwise be rejected by the server today. ## Adding an operation @@ -57,18 +30,10 @@ Adding a System Nexus operation requires generated transfer types, a serialization-context factory, a discoverable protobuf service and method descriptor, and payload-visitor coverage for every nested payload field. -## Language-specific considerations +## Language Specific Considerations ### TypeScript -In TypeScript, creating the protobuf-binary envelope inside the Workflow -isolate is difficult because protobufjs requires writes. The isolate therefore -serializes the envelope with the normal JSON converter. Worker-side payload -processing converts it to protobuf binary before it reaches the server. This -conversion happens regardless of whether a payload codec or external storage -is configured. - -The serialization-context factory is defined for the user model type, while -the transfer type is what remains after leaving the isolate. TypeScript -therefore stores the derived context in `__temporal_system_context` metadata on -the JSON envelope so it can be reused outside the isolate. +In Typescript, it is difficult to actually create the proto binary envelope in the isolate due to read only considerations with the protobufjs library. Instead, the isolate serializes the envelope with the normal json converter, and then processing of that payload during codec/external storage application converts it to proto binary. Notably this happens and needs to happen regardless of the presence of a codec or external storage. + +Additionally, the serialization context is not retrievable outside the isolate because the factory is defined as taking the user model type rather than the transfer type, which is what remains after leaving the isolate. For this reason, the json serialized envelope has an additional `__temporal_system_context` metadata which contains the serialization context so it can be reused outside the isolate.