diff --git a/docs/external-signalling/README.md b/docs/external-signalling/README.md index 3a94a22..3d3ab69 100644 --- a/docs/external-signalling/README.md +++ b/docs/external-signalling/README.md @@ -1,457 +1,112 @@ -# NetherNet External Signalling v1 +# NetherNet External Signalling -NetherNet External Signalling (NXS) lets a NetherNet server use a signalling -provider chosen by its operator. The server registers with the provider and -publishes the information clients need to connect. Each client then brings a -short-lived token that the server can check locally. +NXS lets a NetherNet host use a signalling provider chosen by its operator. +The integration has three flows: register, heartbeat, and accept joins with +asynchronous feedback. This is an experimental contract, updated in place. +The [wire reference](wire-reference.md), [schema](nxs-v1.schema.json) and +[fixtures](nxs-v1.fixtures.json) specify the exact formats under Apache-2.0. -For example, a server can publish its address and certificate fingerprint when -it starts. Later, the provider gives a client those details and a token. The -server checks the token in the client's first packet. It does not need to ask -the provider whether to accept that connection. +## 1. Register with a provider -This is an experimental open specification, identified by -`urn:nethernet:external-signalling:v1`. This document, the -[schema](nxs-v1.schema.json), and the test fixtures define one versioned protocol. -They use this repository's Apache-2.0 license. +Configure the provider's HTTPS origin and fetch +`/.well-known/nethernet-external-signalling`. Check its supported authentication +and use the operation URLs it returns. All URLs must remain on that origin; +never follow redirects when discovering or sending credentials. -## How a connection works +Create and save a P-384 machine key for this instance. Call `register` with the +public key and one of these enrollment choices: -1. The host registers with the provider and proves that it owns its signing key. -2. The host publishes its address, certificate fingerprint, and connection - settings. It sends heartbeats to renew its registration lease. -3. The provider uses that information to give a client a connection answer and - an admission token. -4. The client sends the token in its first STUN packet to the host. -5. The host checks the packet and token, then establishes the connection. The - client's certificate must match the fingerprint in the token. -6. The host reports connection and game outcomes to the provider afterwards. +- `new-service`: create a service and its first instance, using advertised + anonymous proof of work or a bearer token. +- `attach-instance`: add an instance to an existing service, using a bearer + token and authorized placement metadata. -Here, **stateless admission** means that the host needs no saved state for that -client before its first packet arrives. The host still keeps its own keys, -registration, and active connections. A conforming implementation MUST NOT -require a push, poll, shared lookup, offer fetch, or pre-staged client state to -admit a client. Reports about the connection or game outcome never determine -whether the host can accept that first packet. +The provider returns a challenge. Sign its bound proof and call `complete`. +Completion returns the assigned IDs, a fresh process generation, lease deadline +and initial admission key. Save them before publishing readiness. Completion +starts the generation; there is no separate activation call. -NXS covers communication between the host and provider. Account systems, -credential issuance, billing, Microsoft login, DNS management, and the policy -for choosing a host are outside this specification. +On restart, reuse the saved machine key and call `register` with +`{registrationId,protocol,profile}`. Prove the returned challenge through +`complete` to preserve IDs and fence the previous process. If a completion reply +was lost, recover using the saved challenge ID. Repeating a consumed completion +cannot start another generation or reveal its secrets again. -### Terms +Every live replica needs its own key and private state directory. Account/token +issuance, ownership claims and fleet administration belong to the provider. -| Term | Meaning here | -| --- | --- | -| Host or instance | One running NetherNet server. Its instance ID survives a restart. | -| Provider | The service that registers hosts and gives clients connection information. | -| Service | A provider-assigned registration that can contain one or more instances. | -| Lease | The period for which an instance is eligible to receive new connections. Heartbeats renew it. | -| Generation | A counter advanced on activation. Requests from earlier generations are rejected. | -| Host profile | The address, certificate fingerprint, and other settings clients need to connect. | -| Incarnation | A random ID for one bound UDP endpoint. A newly bound endpoint gets a new ID. | -| Admission | Checking a client's token and first packet before creating its native peer. | -| Key epoch | One version of an admission key, identified by `keyId`. | - -ICE checks network reachability using STUN packets. DTLS authenticates and -encrypts the connection. SCTP carries the data channels over that connection. -A UDP tuple identifies a packet's source address and port at a host endpoint. - -## Version and discovery +## 2. Heartbeat to the provider -| Field | v1 value | -| --- | --- | -| Registration/request protocol | `nethernet-external-signalling-v1` | -| Machine request signature | `nxs-es384-v1` | -| Operational profile | `nxs-admission-v1` | -| Discovery path | `/.well-known/nethernet-external-signalling` | -| Stateless capability | `nethernet.stateless-admission.v1` | -| Stateless carrier prefix | `NXS1` | +Send a signed `heartbeat` immediately after startup and whenever its returned +schedule says to check in. The request carries: -### Provider origin and operation URLs +- Health, capacity, load and optional public server status. +- `hostProfile` when endpoint details change; otherwise `hostProfileRevision`. +- `installedKeyIds`, listing installed admission epochs with the active one last. +- Local `state` (`serving`, `draining` or `closed`), the applied provider-state + revision, and whether the integration can report game outcomes. -The configured origin MUST use HTTPS. HTTP is permitted only for loopback -development. Normalize the origin by lowercasing its scheme and host and omitting -default ports. It cannot contain credentials, a path, a query, or a fragment. +The reply returns the accepted profile revision, readiness, lease/schedule, +provider state and any admission-key updates. A host becomes routable only with +a live lease, usable profile and acknowledged installed key. -Fetch discovery with an unauthenticated `GET`. Its `provider` and `controlOrigin` -MUST equal the configured origin. Each operation URL MUST have that same origin -and contain no userinfo or fragment. Clients MUST disable redirects for discovery -and for calls that carry credentials. Sign encoded paths and query strings -exactly as transmitted. +Apply provider state before acknowledging its revision. `draining` stops new +joins and preserves existing sessions; `closed` closes the transport. Provider +routing and credential decisions take effect independently of host check-in. -Discovery contains `provider`, `controlOrigin`, the arrays `protocols`, -`signatures`, `profiles`, and `modes`, an `operations` map, `authorization`, -`limits`, and optional `extensions`. Before sending credentials, clients reject -an unsupported protocol, profile, signature, mode, or required extension. +For a replacement admission key, include a fresh `keyRequestId`. Save and install +the returned key, then immediately heartbeat with the updated profile and +installed IDs. The provider cannot issue new tokens under that epoch before the +acknowledgement. Retain older keys until their reported retirement deadlines. -The [operation table](#operations) defines the operation names. Clients get their -URLs from discovery. `/v1/nxs/` is a recommended path, but providers -can use other paths. +On orderly shutdown, stop accepting new joins and immediately heartbeat with +`state: "draining"`. Do not wait for the periodic timer. A provider outage lets +routing leases expire; it does not by itself close established sessions. -### Authorization and limits +## 3. Accept a stateless join and report the outcome -`authorization` contains `header: "Authorization"` and a `schemes` array. Each -entry has a `scheme` and its supported `modes`: +The provider gives the client the host's connection details and a short-lived +admission token. The client carries that token in its first STUN packet. The +host validates token authentication, expiry, endpoint/client binding and STUN +integrity locally before creating a peer. DTLS must then verify the client +certificate fingerprint from the token. -| Scheme | Allowed modes | -| --- | --- | -| `anonymous-proof-of-work` | `new-service` | -| `bearer-token` | `new-service`, `attach-instance`, or both | +No provider push, poll, lookup or pre-staged client state may gate admission. +The host retains its own background keys and active connection state. -A provider need only advertise the schemes it accepts. It decides how tokens -are issued, what they authorize, and whether they can be reused. Every flow also -requires proof that the instance owns its signing key. +Send signed `outcomes` batches asynchronously, independently of heartbeat timing: -| Limit | v1 constraint | +| Observation | Required feedback | | --- | --- | -| `maxBodyBytes` | At most 65536 | -| `clockSkewMs` | At most 60000 | -| `heartbeatIntervalMs` | 1000–30000 | -| `leaseMs` | Advertised lease duration | -| `maxControlPage` | At most 100 | - -`checkInVersion: 1` enables the provider to set the next check-in time in its -response. A provider MUST advertise every limit it enforces, reject oversized -bodies, and return errors as `{"code":"lowercase_machine_code"}` with an -appropriate HTTP failure status. Clients limit response size before parsing. - -On a transient transport failure or HTTP 429, 502, 503, or 504, the supplied -client makes at most three attempts in total. Retries use exponential delays -with jitter and an upper bound. A `Retry-After` value over ten seconds returns -a retry-later result. Retrying never extends a lease or challenge expiry. - -## Registration and persistent identity - -### Save the instance key - -Generate a fresh P-384 machine signing key for each logical instance. Save it -before requesting a challenge. A restart reuses that instance's saved state; -live replicas cannot share a key or state directory. Images and templates MUST -contain neither machine identity nor DTLS private keys. - -Clients lock their state directory and write private state atomically with -owner-only permissions. Sync both files and directories to durable storage. -If saving state fails, stop advertising healthy readiness. - -### Request a challenge - -The request contains `protocol`, `mode`, `profile`, `publicKeyJwk`, explicit -`authorization: {scheme}`, and optional `label` and `placement`. - -Send a bearer credential only to the challenge operation, in -`Authorization: Bearer `. It MUST NOT appear in JSON, proofs, saved state, -or logs. `attach-instance` requires both bearer authorization and placement. -The token authorizes access to the service; a client-provided label grants no -permission. - -Placement is `{region,pool,tags?}`: - -| Field | Constraint | -| --- | --- | -| `region` | Immutable routing label matching `[A-Za-z0-9_-]{1,32}` | -| `pool` | Immutable routing label matching `[A-Za-z0-9_-]{1,64}` | -| `tags` | At most 16 keys matching `[A-Za-z0-9_.-]{1,32}`; values are trimmed strings of 1–64 characters with no control characters | - -The challenge binds the exact placement. At completion, the provider rechecks -that the token authorizes it as part of the same atomic operation that creates -the registration. These fields do not prescribe how a provider selects a host. - -The public JWK is EC/P-384. Its `x` and `y` values use canonical, unpadded -base64url and each encode exactly 48 bytes. It MUST NOT contain `d`. The RFC 7638 -thumbprint is SHA-256 of UTF-8 JSON with members in this exact order: -`crv,kty,x,y`. ES384 signatures use the 96-byte IEEE-P1363 form `r || s`, encoded -as unpadded base64url. Reject DER signatures and noncanonical base64url. - -The challenge response contains `protocol`, `signature`, `challengeId`, `nonce`, -`audience`, `thumbprint`, `context`, `contextDigest`, `expiresAt`, `serverTime`, -and `pow: {algorithm:"sha256-leading-zero-bits-v0",difficulty}`. - -Proof-of-work difficulty is 0–24. Bearer-authorized and recovery flows use zero. -An authorization reference is an opaque identifier, never the credential itself. -Expiry and server times are integer epoch milliseconds. - -### Complete registration - -Canonical arrays use UTF-8 JSON with no whitespace or Unicode normalization. -Use an empty string for a missing context string. `contextDigest` is the -unpadded base64url SHA-256 digest of: - -```text -[mode,profile,label,authorizationId,serviceId,region,pool,registrationId] -``` - -When tags are nonempty, append `tagsDigest` to that array. Compute `tagsDigest` -in the same way from sorted `[key,value]` pairs. The completion proof is: - -```text -[protocol,"complete",audience,challengeId,nonce,thumbprint,contextDigest, - expiresAt,proofNonce,idempotencyKey] -``` - -Proof of work counts the leading zero bits in SHA-256 of those bytes. Send -`protocol,challengeId,proofNonce,idempotencyKey,signature` to complete registration. -The provider MUST check expiry, binding, signature, difficulty, current authority, -and single-use completion atomically with resource creation. - -Retrying completion MUST NOT return one-time key secrets again. If completion -was interrupted, recover the registration by proving ownership of the same key. - -Completion returns `protocol,provider,registrationId,serviceId,instanceId,keyId, -profile,publicAddress,placement,heartbeatIntervalMs,leaseGeneration,leaseDeadline, -readiness`, plus optional one-time `ticketKey` and `extensions`. Save the IDs and -key material before activation. Remove secrets from registration results exposed -to applications and from diagnostic output. - -## Signed lifecycle and host profile - -### Sign operational requests - -Use the registered machine key for every operational request. The enrollment -bearer token is used only for the challenge request. - -Required headers are `nxs-instance-id`, `nxs-key-id`, `nxs-timestamp`, -`nxs-signature-version`, `nxs-generation`, `nxs-sequence`, `nxs-signature`, and -`idempotency-key`. The timestamp is epoch milliseconds. Generation and sequence -are nonnegative integers. Save each reserved sequence number before sending its -request. The signature covers this array: - -```text -[protocol,signatureVersion,audience,method,encodedPathAndQuery,timestamp, - instanceId,keyId,idempotencyKey,generation,sequence,base64url(sha256(bodyBytes))] -``` - -For an empty body, hash a zero-length byte sequence. Providers reject stale -generations, reused sequence numbers, invalid timestamps, and invalid signatures. -An idempotent retry can return the recorded result, with secrets removed, if its -intent and semantic request are unchanged. It cannot apply the operation again. - -Activation increments the generation and resets the sequence. The provider then -rejects requests from the old process. Signed state-changing operations must use -the active profile. To change profiles, recover the registration and send a -signed activation request. - -### Operations - -| Operation | Request | Required result or behavior | -| --- | --- | --- | -| `challenges` | POST challenge request, optional bearer | Challenge bound to the registration request | -| `complete` | POST completion proof | New or recovered registration; return secrets only once | -| `recover` | POST `{registrationId,protocol,profile}` | Challenge for the current or pending machine key; preserve assigned IDs | -| `activate` | Signed POST `{profile}` | Increment `leaseGeneration`, return `leaseDeadline`, and reset stale host readiness | -| `readiness` | Signed GET | Whether the host can receive new connections, with reasons and optional extension metadata | -| `host-profile` | Signed POST profile below | A `revision` cannot change once published; updates use a higher revision. Reject unusable candidates or keys | -| `heartbeat` | Signed POST health/status below | Receipt time, renewed lease, and optional check-in schedule | -| `control` | Signed GET, optional cursor | Limited `commands` page, optional `cursor`, and `serverTime` | -| `control/ack` | Signed POST `{cursor}` | Acknowledge only lifecycle commands that have finished | -| `ticket-keys` | Signed POST `{}` | One-time `{ticketKey:{keyId,secret,...}}` for a new key epoch | -| `ticket-keys/ack` | Signed POST `{keyId}` | Confirm the key is installed before using its epoch for new connections | -| `ticket-events` / `events` | Signed POST `{events:[...]}` | Limited batches of asynchronous observations; retries do not duplicate them | -| `rotate` | Signed POST `{publicKeyJwk,proof}` | New `keyId` after proof of ownership of the replacement key | -| `retire` | Signed POST `{keyId}` | Retire the previous machine signing key | -| `drain` | Signed POST `{}` | Stop directing and accepting new connections; preserve existing sessions | -| `deregister` | Signed POST `{}` | Stop directing connections to the instance and end its registration | - -### Rotate a machine key - -The rotation proof bytes are `[protocol,"rotate",audience,instanceId,oldKeyId, -newThumbprint,generation,idempotencyKey]`. Save the replacement private key before -requesting rotation. Save the result before retiring the old key. After an -interrupted rotation, recovery can use the provider's returned key thumbprint -to identify which key is current. - -### Publish the host profile +| Both data channels become usable | `ticket.data_channels_open` | +| An authenticated observed attempt fails before transport becomes usable | `ticket.failed`, with a bounded reason | +| The game admits or rejects the player | `ticket.game_joined` or `ticket.game_rejected`, when the integration observes this boundary | -`host-profile` contains `candidates`, `dtlsFingerprint`, `credentialKeyId`, -`sctpPort`, `maxMessageSize`, and `statelessAdmission: {capability,incarnation}`. -Generate a fresh random 16-byte `incarnation`, encoded as lowercase hex, for -each bound native endpoint. The fingerprint is `sha-256 ` followed by the -certificate's digest bytes in colon-separated uppercase hex. +Declare game-outcome support as `available` or `unavailable` in heartbeat. A +transport connection never proves successful gameplay. Intermediate ICE, DTLS +and SCTP stages are optional diagnostics in the same stream. -Each candidate contains `foundation,component,protocol,priority,address,port,type`. -Publish only reachable UDP candidates that are explicitly chosen for advertisement. -The bind address and the advertised address serve different purposes. A host can -bind to all interfaces, but it cannot advertise wildcard `0.0.0.0` or `::`. -The deployment or provider must establish reachability through NAT or a relay; -a passing registration test does not prove that clients can reach the address. +Each event contains `ticketId`, `stage`, `occurredAt` and optional `reason`. +Retry bounded batches without duplicating observations. Do not send player +identity, SDP, credentials or game payloads. Reporting failure never delays +admission or renews a lease. Missing feedback means an unknown outcome: a client +may never reach the host, or the host may crash before reporting. -Prepare the host's DTLS certificate and key before publishing its profile. Keep -the private key local. All peers using that profile use that certificate, so -clients see the fingerprint the provider advertised. The host may use a new -certificate for a later endpoint incarnation after publishing its new fingerprint. -A permanent certificate shared across a fleet is neither required nor advised. +## Operation reference -Three types of key have separate jobs: +All operations use POST; all except `register` and `complete` use the machine +request signature. URLs come from discovery. -| Key | Purpose | +| Operation | Purpose | | --- | --- | -| Machine signing key | Authenticate the host's requests to the provider | -| DTLS certificate and private key | Authenticate the host during the client connection | -| Admission key | Protect and validate the client's admission token | - -### Install admission keys - -Each key has a `keyId` of four uppercase alphanumeric characters, a `secret` of -32–256 UTF-8 characters, and optional `notBefore` and `retireAfter` times in epoch -milliseconds. Install at most eight epochs atomically and acknowledge them. Then -publish a profile that uses an active, installed epoch. - -Reject tokens before the key's activation time or after its retirement time. -Erase retired key material. Rotating keys does not extend token expiry. - -### Send heartbeats and report readiness - -A heartbeat contains `healthy,capacity,load,protocolVersion,build,hostProfileRevision, -clockUnixMillis` and optional `region,serverStatus,checkInVersion`. Capacity and -load describe routing capacity; they are independent of the advertised player -and maximum-player counts. Status contains -`name,protocol,version,level,players,maxPlayers,gameType`. A failed publication -does not refresh the timestamp of previously published status. - -A host is ready to receive connections only when it has a current identity and -generation, a live lease, a usable fresh host profile, and acknowledged installed -keys. Optional product extensions cannot affect this core readiness check. - -With check-in v1, the heartbeat response contains ISO8601 `receivedAt` and: - -```text -checkIn: {version:1,afterMillis,nextCheckInAt,leaseExpiresAt,minUpdateIntervalMillis, - controlPollAfterMillis} -``` - -Absolute times in `checkIn` are epoch milliseconds. `nextCheckInAt` is before -lease expiry. Hosts use monotonic clocks for scheduling and include network time -in the interval. Changed activity or status can trigger an earlier heartbeat, -subject to the rate limit. On restart, publish immediately and discard the old -schedule. If the provider is unavailable, routing leases expire; existing sessions -are not closed solely because of that outage. - -### Handle controls and report outcomes - -This profile supports `noop,drain,suspend,revoke`. Do not silently acknowledge an -unknown control. An unknown command can prevent advancing the page cursor, but -later known lifecycle commands still need processing. `join-admission` is not a -v1 control; accepting a client never waits for that command. - -Event batches contain at most 100 entries. Keep only redacted correlation data, -stage or type, timestamp, and reason fields with size limits. Never send SDP, -private keys, player identity, or game payloads as telemetry. A working transport -connection is a separate outcome from `ticket.game_joined` (ready to play) or -`ticket.game_rejected`. - -## Stateless admission carrier - -### Carry the token in the ICE username - -The client's first STUN USERNAME is `:`, where: - -```text -answerUfrag = "NXS1" + keyId + unpaddedBase64(nonce || ciphertext || tag) -``` - -Use the standard base64 alphabet, including `+` and `/`, which ICE permits. -Do not use base64url. The total ufrag length is at most 256 characters. Before -allocating peer state, reject noncanonical encoding, trailing padding, a wrong -prefix, unknown key epochs, and oversized input. - -AES-256-GCM uses a random 12-byte nonce and a 16-byte tag. Its key is -`HMAC-SHA256(secret, "nxs-stateless-aead-v1" || NUL || audience)`. -The audience is `nxs-stateless-host-v1/`. The additional authenticated -data (AAD) is -`"nxs-stateless-admission-v1" || NUL || ("NXS1"+keyId) || NUL || audience || NUL || clientUfrag`. - -| Plaintext offset | Size | Meaning, unsigned big-endian where numeric | -| --- | --- | --- | -| 0 | 4 | Expiry in epoch seconds, exactly representable in milliseconds | -| 4 | 32 | SHA-256 client certificate fingerprint | -| 36 | 2 | Client SCTP port, 1–65535 | -| 38 | 4 | Client maximum message size, 1–262144 | -| 42 | 16 | Opaque caller-context hash, no account-specific interpretation | -| 58 | 8 | NetherNet network ID, unsigned 64-bit | -| 66 | 1 | Client ICE password length, 22–91 | -| 67 | N | Client ICE password in ICE base64 alphabet | - -The host's local ICE password is the unpadded standard base64 encoding of the -first 24 bytes of -`HMAC-SHA256(secret, "nxs-stateless-ice-v1" || NUL || audience || NUL || answerUfrag)`. -The ticket correlation ID is the first 16 bytes of SHA-256 of the ASCII answer -ufrag, encoded as lowercase hex. - -### Validate the first packet - -A token can be valid for at most 120 seconds. Before assigning the UDP tuple to a -peer or allocating a peer connection, the host checks expiry, field bounds, GCM -authentication, client binding, and STUN MESSAGE-INTEGRITY. The DTLS handshake -MUST then verify the client fingerprint from the token. - -Only a retransmission of the identical token from the same UDP tuple can reuse -a reservation. Reject the same token from another tuple. Also reject a conflicting -admission on an occupied tuple. - -Limit the number of sessions, pending handshakes, used-token records, queued -validation tasks, and retained requests. Duplicate requests for the same pending -attempt share one decision. Preserve enough of the first request to respond after -acceptance: completing admission MUST NOT depend on the client retransmitting. -Release admission capacity only after the connection's resources have been -released. A failed integrity check MUST NOT consume the token, since a copied -token alone does not prove that the sender has its ICE password. - -#### Reference implementation - -The supplied Network implementation uses a 60-second token limit. libjuice -retains the first STUN packet and sends parsed metadata to Java for asynchronous -token validation. libdatachannel verifies STUN integrity before creating the -peer, outside the UDP receive lock. It then processes the retained request after -the application has installed its callbacks. Established transport packets stay -native, and capacity remains reserved until native teardown finishes. - -Other implementations may meet the requirements above using different languages, -threading models, and transport libraries. - -## Optional extensions and compatibility - -### Extensions - -Providers can add optional application metadata without making it part of NXS. -For example, a product could supply an account-claim link. NXS does not define -what claiming an account means or require other providers to implement it. - -`extensions` is an object with at most 16 reverse-DNS namespace keys, such as -`com.example.feature`, and at most 16384 bytes of encoded UTF-8 JSON. Keys use -lowercase domain-style labels and have at most 128 characters. Each value is -`{version:positiveInteger,critical:boolean,data:object}`. - -Pass through or ignore unknown optional extensions; never execute them -automatically. Reject unsupported critical extensions before sending credentials -or activating. An optional extension cannot change the core protocol rules. -TLS and request signatures still authenticate bodies and operation paths. - -An extension can advertise URLs in `data.operations`. An application can request -one of these operations only after validating the namespace, version, and meaning. -The generic transport still requires the same provider origin and signs the -exact path. - -### Upgrade and rollback - -Recover saved IDs and keys into this profile with -`recover {registrationId,protocol,profile}`, then signed `activate {profile}`. -Verify the same key and origin, preserve IDs and DTLS files, and record the new -profile and generation atomically. Legacy protocol bytes MUST NOT be relabelled -as v1. Providers may keep separately negotiated legacy adapters; the neutral -Java module implements only NXS. - -Rollback uses the previous client with explicit recovery and signed profile -activation. Never bypass machine authentication or copy a live state directory. - -## Conformance - -Run `node docs/external-signalling/fixtures.mjs` to verify the independent -JavaScript signing, encryption, and fixture hashes. `--write` regenerates public -test signatures. The JVM suites load these same files through Gradle resources. - -The independent test provider covers registration, signed operations, status, -keys, outcomes, drain, and recovery without a product account system. Native -tests separately check raw STUN admission and DTLS transport. - -Report stock-client admission, gameplay, and routing across two hosts separately -from fixture and native tests. Passing those tests does not prove that a stock -client can join and play. +| `register` | Request an enrollment or recovery challenge | +| `complete` | Prove the challenge and start the process generation | +| `heartbeat` | Exchange host health, profile, keys, lifecycle state and readiness | +| `outcomes` | Report transport and game observations | +| `rotate` | Prove and install a replacement machine signing key | +| `retire` | Retire the previous machine signing key | +| `deregister` | Permanently end this instance's registration | + +Machine-key maintenance is separate from admission-key updates. Exact signing, +request fields, key handling, token layout, bounds and retries are in the +[wire reference](wire-reference.md). diff --git a/docs/external-signalling/nxs-v1.schema.json b/docs/external-signalling/nxs-v1.schema.json index b003b4e..36ce731 100644 --- a/docs/external-signalling/nxs-v1.schema.json +++ b/docs/external-signalling/nxs-v1.schema.json @@ -184,6 +184,49 @@ }, "extensions": { "$ref": "#/$defs/extensions" + }, + "operations": { + "type": "object", + "required": [ + "register", + "complete", + "heartbeat", + "outcomes", + "rotate", + "retire", + "deregister" + ], + "properties": { + "register": { + "type": "string", + "format": "uri" + }, + "complete": { + "type": "string", + "format": "uri" + }, + "heartbeat": { + "type": "string", + "format": "uri" + }, + "outcomes": { + "type": "string", + "format": "uri" + }, + "rotate": { + "type": "string", + "format": "uri" + }, + "retire": { + "type": "string", + "format": "uri" + }, + "deregister": { + "type": "string", + "format": "uri" + } + }, + "additionalProperties": false } } }, @@ -290,17 +333,6 @@ "placement" ] }, - "activation": { - "type": "object", - "required": [ - "profile" - ], - "properties": { - "profile": { - "const": "nxs-admission-v1" - } - } - }, "rotation": { "type": "object", "required": [ @@ -315,14 +347,6 @@ "secret" ] }, - "pendingAction": { - "type": "object", - "required": [ - "url", - "expiresAt", - "text" - ] - }, "extensions": { "type": "object", "maxProperties": 16, @@ -373,6 +397,396 @@ "const": "nxs-admission-v1" } } + }, + "register": { + "oneOf": [ + { + "$ref": "#" + }, + { + "$ref": "#/$defs/recovery" + } + ] + }, + "hostState": { + "enum": [ + "serving", + "draining", + "closed" + ] + }, + "hostProfile": { + "type": "object", + "required": [ + "candidates", + "dtlsFingerprint", + "credentialKeyId", + "sctpPort", + "maxMessageSize", + "statelessAdmission" + ], + "properties": { + "candidates": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "object", + "required": [ + "foundation", + "component", + "protocol", + "priority", + "address", + "port", + "type" + ], + "properties": { + "foundation": { + "type": "string", + "pattern": "^[A-Za-z0-9._:-]{1,32}$" + }, + "component": { + "const": 1 + }, + "protocol": { + "const": "udp" + }, + "priority": { + "type": "integer", + "minimum": 1, + "maximum": 2147483647 + }, + "address": { + "type": "string", + "maxLength": 253 + }, + "port": { + "type": "integer", + "minimum": 1, + "maximum": 65535 + }, + "type": { + "enum": [ + "host", + "srflx", + "relay" + ] + } + } + } + }, + "dtlsFingerprint": { + "type": "string", + "pattern": "^sha-256 [0-9A-F]{2}(?::[0-9A-F]{2}){31}$" + }, + "credentialKeyId": { + "type": "string", + "pattern": "^[A-Z0-9]{4}$" + }, + "sctpPort": { + "const": 5000 + }, + "maxMessageSize": { + "const": 262144 + }, + "statelessAdmission": { + "type": "object", + "required": [ + "capability", + "incarnation" + ], + "properties": { + "capability": { + "const": "nethernet.stateless-admission.v1" + }, + "incarnation": { + "type": "string", + "pattern": "^[0-9a-f]{32}$" + } + } + } + } + }, + "heartbeat": { + "type": "object", + "required": [ + "healthy", + "capacity", + "load", + "protocolVersion", + "clockUnixMillis", + "checkInVersion", + "state", + "appliedStateRevision", + "gameOutcomes" + ], + "properties": { + "healthy": { + "type": "boolean" + }, + "capacity": { + "type": "integer", + "minimum": 0, + "maximum": 1000000 + }, + "load": { + "type": "number", + "minimum": 0, + "maximum": 1 + }, + "protocolVersion": { + "type": "string", + "maxLength": 128 + }, + "clockUnixMillis": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "checkInVersion": { + "const": 1 + }, + "state": { + "$ref": "#/$defs/hostState" + }, + "appliedStateRevision": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "gameOutcomes": { + "enum": [ + "available", + "unavailable" + ] + }, + "build": { + "type": "string", + "maxLength": 128 + }, + "region": { + "type": "string", + "maxLength": 32 + }, + "serverStatus": { + "type": "object" + }, + "hostProfile": { + "$ref": "#/$defs/hostProfile" + }, + "hostProfileRevision": { + "type": "string", + "maxLength": 184 + }, + "installedKeyIds": { + "type": "array", + "maxItems": 8, + "uniqueItems": true, + "items": { + "type": "string", + "pattern": "^[A-Z0-9]{4}$" + } + }, + "keyRequestId": { + "type": "string", + "pattern": "^[A-Za-z0-9_-]{16,128}$" + }, + "extensions": { + "$ref": "#/$defs/extensions" + } + } + }, + "checkIn": { + "type": "object", + "required": [ + "version", + "afterMillis", + "nextCheckInAt", + "leaseExpiresAt", + "minUpdateIntervalMillis" + ], + "properties": { + "version": { + "const": 1 + }, + "afterMillis": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "nextCheckInAt": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "leaseExpiresAt": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "minUpdateIntervalMillis": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + } + } + }, + "heartbeatResponse": { + "type": "object", + "required": [ + "accepted", + "receivedAt", + "staleAfter", + "hostProfileRevision", + "activeKeyId", + "leaseGeneration", + "readiness", + "desiredState", + "retirements" + ], + "properties": { + "accepted": { + "const": true + }, + "receivedAt": { + "type": "string", + "format": "date-time" + }, + "staleAfter": { + "type": "string", + "format": "date-time" + }, + "hostProfileRevision": { + "type": [ + "string", + "null" + ] + }, + "activeKeyId": { + "type": "string", + "pattern": "^[A-Z0-9]{4}$" + }, + "leaseGeneration": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "readiness": { + "$ref": "#/$defs/readiness" + }, + "desiredState": { + "type": "object", + "required": [ + "revision", + "state" + ], + "properties": { + "revision": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "state": { + "$ref": "#/$defs/hostState" + } + } + }, + "checkIn": { + "$ref": "#/$defs/checkIn" + }, + "keyRequest": { + "type": "object", + "required": [ + "id", + "keyId" + ], + "properties": { + "id": { + "type": "string", + "maxLength": 128 + }, + "keyId": { + "type": "string", + "pattern": "^[A-Z0-9]{4}$" + } + } + }, + "ticketKey": { + "$ref": "#/$defs/ticketKey" + }, + "retirements": { + "type": "array", + "items": { + "type": "object", + "required": [ + "keyId", + "retireAfter" + ], + "properties": { + "keyId": { + "type": "string", + "pattern": "^[A-Z0-9]{4}$" + }, + "retireAfter": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + } + } + } + } + } + }, + "outcomes": { + "type": "object", + "required": [ + "events" + ], + "properties": { + "events": { + "type": "array", + "minItems": 1, + "maxItems": 100, + "items": { + "type": "object", + "required": [ + "ticketId", + "stage", + "occurredAt" + ], + "properties": { + "ticketId": { + "type": "string", + "minLength": 1, + "maxLength": 256 + }, + "stage": { + "enum": [ + "ticket.ice_seen", + "ticket.ice_connected", + "ticket.dtls_connected", + "ticket.sctp_connected", + "ticket.data_channels_open", + "ticket.game_joined", + "ticket.game_rejected", + "ticket.failed" + ] + }, + "occurredAt": { + "type": "string", + "format": "date-time" + }, + "reason": { + "type": "string", + "maxLength": 128 + } + }, + "additionalProperties": false + } + } + }, + "additionalProperties": false } }, "x-context-order": [ @@ -390,7 +804,7 @@ ], "x-signature-format": "ES384 P1363 r||s 96 bytes unpadded base64url", "x-canonicalization": "UTF-8 JSON arrays, no whitespace, no Unicode normalization, epoch milliseconds as integers, absent context strings are empty", - "description": "Canonical registration and lifecycle metadata for the nxs-admission-v1 profile.", + "description": "Registration, heartbeat and stateless join feedback for the experimental NXS contract.", "x-profile": "nxs-admission-v1", "x-discovery-path": "/.well-known/nethernet-external-signalling", "x-headers": [ @@ -404,22 +818,12 @@ "idempotency-key" ], "x-operations": [ - "challenges", + "register", "complete", - "recover", - "activate", - "drain", - "deregister", + "heartbeat", + "outcomes", "rotate", "retire", - "ticket-keys", - "ticket-keys/ack", - "readiness", - "heartbeat", - "host-profile", - "control", - "control/ack", - "ticket-events", - "events" + "deregister" ] } diff --git a/docs/external-signalling/wire-reference.md b/docs/external-signalling/wire-reference.md new file mode 100644 index 0000000..06811c7 --- /dev/null +++ b/docs/external-signalling/wire-reference.md @@ -0,0 +1,419 @@ +# NXS wire reference + +This is the normative format reference for the [three integration flows](README.md). +Protocol identifiers describe the current experimental format; this revision +replaces its earlier operation surface in place. + +## Version and discovery + +| Field | v1 value | +| --- | --- | +| Registration/request protocol | `nethernet-external-signalling-v1` | +| Machine request signature | `nxs-es384-v1` | +| Operational profile | `nxs-admission-v1` | +| Discovery path | `/.well-known/nethernet-external-signalling` | +| Stateless capability | `nethernet.stateless-admission.v1` | +| Stateless carrier prefix | `NXS1` | + +### Provider origin and operation URLs + +The configured origin MUST use HTTPS. HTTP is permitted only for loopback +development. Normalize the origin by lowercasing its scheme and host and omitting +default ports. It cannot contain credentials, a path, a query, or a fragment. + +Fetch discovery with an unauthenticated `GET`. Its `provider` and `controlOrigin` +MUST equal the configured origin. Each operation URL MUST have that same origin +and contain no userinfo or fragment. Clients MUST disable redirects for discovery +and for calls that carry credentials. Sign encoded paths and query strings +exactly as transmitted. + +Discovery contains `provider`, `controlOrigin`, the arrays `protocols`, +`signatures`, `profiles`, and `modes`, an `operations` map, `authorization`, +`limits`, and optional `extensions`. Before sending credentials, clients reject +an unsupported protocol, profile, signature, mode, or required extension. + +The [operation table](README.md#operation-reference) defines the operation names. Clients get their +URLs from discovery. `/v1/nxs/` is a recommended path, but providers +can use other paths. + +### Authorization and limits + +`authorization` contains `header: "Authorization"` and a `schemes` array. Each +entry has a `scheme` and its supported `modes`: + +| Scheme | Allowed modes | +| --- | --- | +| `anonymous-proof-of-work` | `new-service` | +| `bearer-token` | `new-service`, `attach-instance`, or both | + +A provider need only advertise the schemes it accepts. It decides how tokens +are issued, what they authorize, and whether they can be reused. Every flow also +requires proof that the instance owns its signing key. + +| Limit | v1 constraint | +| --- | --- | +| `maxBodyBytes` | At most 65536 | +| `clockSkewMs` | At most 60000 | +| `heartbeatIntervalMs` | 1000–30000 | +| `leaseMs` | Advertised lease duration | + +`checkInVersion: 1` enables the provider to set the next check-in time in its +response. A provider MUST advertise every limit it enforces, reject oversized +bodies, and return errors as `{"code":"lowercase_machine_code"}` with an +appropriate HTTP failure status. Clients limit response size before parsing. + +On a transient transport failure or HTTP 429, 502, 503, or 504, the supplied +client makes at most three attempts in total. Retries use exponential delays +with jitter and an upper bound. A `Retry-After` value over ten seconds returns +a retry-later result. Outcome uploads use one attempt with a three-second timeout +and a ten-second failure backoff so they cannot starve heartbeat renewal. +Retrying never extends a lease or challenge expiry. + +## Registration and persistent identity + +### Save the instance key + +Generate a fresh P-384 machine signing key for each logical instance. Save it +before requesting a challenge. A restart reuses that instance's saved state; +live replicas cannot share a key or state directory. Images and templates MUST +contain neither machine identity nor DTLS private keys. + +Clients lock their state directory and write private state atomically with +owner-only permissions. Sync both files and directories to durable storage. +If saving state fails, stop advertising healthy readiness. + +### `register` request + +The request contains `protocol`, `mode`, `profile`, `publicKeyJwk`, explicit +`authorization: {scheme}`, and optional `label` and `placement`. + +Send a bearer credential only to the enrollment `register` operation, in +`Authorization: Bearer `. It MUST NOT appear in JSON, proofs, saved state, +or logs. `attach-instance` requires both bearer authorization and placement. +The token authorizes access to the service; a client-provided label grants no +permission. + +Placement is `{region,pool,tags?}`: + +| Field | Constraint | +| --- | --- | +| `region` | Immutable routing label matching `[A-Za-z0-9_-]{1,32}` | +| `pool` | Immutable routing label matching `[A-Za-z0-9_-]{1,64}` | +| `tags` | At most 16 keys matching `[A-Za-z0-9_.-]{1,32}`; values are trimmed strings of 1–64 characters with no control characters | + +The challenge binds the exact placement. At completion, the provider rechecks +that the token authorizes it as part of the same atomic operation that creates +the registration. These fields do not prescribe how a provider selects a host. + +The public JWK is EC/P-384. Its `x` and `y` values use canonical, unpadded +base64url and each encode exactly 48 bytes. It MUST NOT contain `d`. The RFC 7638 +thumbprint is SHA-256 of UTF-8 JSON with members in this exact order: +`crv,kty,x,y`. ES384 signatures use the 96-byte IEEE-P1363 form `r || s`, encoded +as unpadded base64url. Reject DER signatures and noncanonical base64url. + +The challenge response contains `protocol`, `signature`, `challengeId`, `nonce`, +`audience`, `thumbprint`, `context`, `contextDigest`, `expiresAt`, `serverTime`, +and `pow: {algorithm:"sha256-leading-zero-bits-v0",difficulty}`. + +Proof-of-work difficulty is 0–24. Bearer-authorized and recovery flows use zero. +An authorization reference is an opaque identifier, never the credential itself. +Expiry and server times are integer epoch milliseconds. + +### `complete` request + +Canonical arrays use UTF-8 JSON with no whitespace or Unicode normalization. +Use an empty string for a missing context string. `contextDigest` is the +unpadded base64url SHA-256 digest of: + +```text +[mode,profile,label,authorizationId,serviceId,region,pool,registrationId] +``` + +When tags are nonempty, append `tagsDigest` to that array. Compute `tagsDigest` +in the same way from sorted `[key,value]` pairs. The completion proof is: + +```text +[protocol,"complete",audience,challengeId,nonce,thumbprint,contextDigest, + expiresAt,proofNonce,idempotencyKey] +``` + +Proof of work counts the leading zero bits in SHA-256 of those bytes. Send +`protocol,challengeId,proofNonce,idempotencyKey,signature` to complete registration. +The provider MUST check expiry, binding, signature, difficulty, current authority, +and single-use completion atomically with resource creation. + +Retrying completion MUST NOT return one-time key secrets again. If completion +was interrupted, recover the registration by proving ownership of the same key. + +Completion returns `protocol,provider,registrationId,serviceId,instanceId,keyId, +profile,publicAddress,placement,heartbeatIntervalMs,leaseGeneration,leaseDeadline, +readiness`, plus optional one-time `ticketKey` and `extensions`. Completion atomically starts a new generation, clears previous readiness and +resets the operational sequence to zero. Save the IDs and key material before +heartbeat. Recovery uses `register {registrationId,protocol,profile}` and the +same completion proof. A deregistered instance cannot recover. Remove secrets from registration results exposed +to applications and from diagnostic output. + +## Signed requests and machine-key maintenance + +### Sign operational requests + +Use the registered machine key for every operational request. The enrollment +bearer token is used only for the challenge request. + +Required headers are `nxs-instance-id`, `nxs-key-id`, `nxs-timestamp`, +`nxs-signature-version`, `nxs-generation`, `nxs-sequence`, `nxs-signature`, and +`idempotency-key`. The timestamp is epoch milliseconds. Generation and sequence +are nonnegative integers. Save each reserved sequence number before sending its +request. The signature covers this array: + +```text +[protocol,signatureVersion,audience,method,encodedPathAndQuery,timestamp, + instanceId,keyId,idempotencyKey,generation,sequence,base64url(sha256(bodyBytes))] +``` + +For an empty body, hash a zero-length byte sequence. Providers reject stale +generations, reused sequence numbers, invalid timestamps, and invalid signatures. +An idempotent retry can return the recorded result, with secrets removed, if its +intent and semantic request are unchanged. It cannot apply the operation again. + +Each completed registration/recovery starts a generation exactly once. The +provider rejects writes from previous generations. A replay cannot advance the +generation or extend the original lease. Replaying a consumed completion returns +a recovery-required error; recover with a fresh challenge. + +### Rotate a machine key + +The rotation proof bytes are `[protocol,"rotate",audience,instanceId,oldKeyId, +newThumbprint,generation,idempotencyKey]`. Save the replacement private key before +requesting rotation. Save the result before retiring the old key. After an +interrupted rotation, recovery can use the provider's returned key thumbprint +to identify which key is current. + +`retire` carries `{keyId}` and must be signed by a different, current machine +key. `deregister` carries `{}` and permanently ends registration. Neither is an +admission-key rotation or an ordinary graceful drain. + +## `heartbeat` + +Required fields: `healthy,capacity,load,protocolVersion,clockUnixMillis, +checkInVersion,state,appliedStateRevision,gameOutcomes`. +Optional fields: `build,region,serverStatus,hostProfile,hostProfileRevision, +installedKeyIds,keyRequestId,extensions`. + +- `capacity` is an integer from 0 to 1000000; `load` is a finite number from 0 to 1. +- `state` is `serving`, `draining` or `closed`. A draining endpoint cannot resume + serving in the same generation; a fresh endpoint requires recovery/completion. +- `gameOutcomes` is `available` when the integration observes game acceptance and + rejection, otherwise `unavailable`. +- `appliedStateRevision` is a nonnegative integer. A response carries + `desiredState: {revision,state}`. Reject unknown states or regressing revisions; + acknowledge only state that finished applying. Pending application triggers + a bounded earlier heartbeat. Receipt alone is not acknowledgement. +- `clockUnixMillis` is an increasing snapshot clock within the generation and + must be within 30000 milliseconds of provider time. +- `region` cannot change authorized placement. `serverStatus` contains + `name,protocol,version,level,players,maxPlayers,gameType`; it is independent of + routing capacity/load. Omitted or failed status publication does not refresh + a previous status snapshot. + +### Publish the host profile + +`heartbeat.hostProfile` contains `candidates`, `dtlsFingerprint`, `credentialKeyId`, +`sctpPort`, `maxMessageSize`, and `statelessAdmission: {capability,incarnation}`. +Generate a fresh random 16-byte `incarnation`, encoded as lowercase hex, for +each bound native endpoint. The fingerprint is `sha-256 ` followed by the +certificate's digest bytes in colon-separated uppercase hex. + +Each candidate contains `foundation,component,protocol,priority,address,port,type`. +Publish 1–32 candidates. Foundations match `[A-Za-z0-9._:-]{1,32}`; component is +1, protocol is `udp`, priority is 1–2147483647, port is 1–65535, and type is +`host`, `srflx` or `relay`. Addresses are IP literals. +Publish only reachable UDP candidates that are explicitly chosen for advertisement. +The bind address and the advertised address serve different purposes. A host can +bind to all interfaces, but it cannot advertise wildcard `0.0.0.0` or `::`. +The deployment or provider must establish reachability through NAT or a relay; +a passing registration test does not prove that clients can reach the address. + +Prepare the host's DTLS certificate and key before publishing its profile. Keep +the private key local. All peers using that profile use that certificate, so +clients see the fingerprint the provider advertised. The host may use a new +certificate for a later endpoint incarnation after publishing its new fingerprint. +A permanent certificate shared across a fleet is neither required nor advised. + +Three types of key have separate jobs: + +| Key | Purpose | +| --- | --- | +| Machine signing key | Authenticate the host's requests to the provider | +| DTLS certificate and private key | Authenticate the host during the client connection | +| Admission key | Protect and validate the client's admission token | + +The provider assigns `hostProfileRevision` in its reply. Send that revision +on later heartbeats until the profile changes. A request retry returns the same +revision. Profile publication, acknowledgement of its installed key, and the +lease update must commit consistently. A profile using an uninstalled epoch +cannot become routable. + +### Admission-key exchange + +`installedKeyIds` contains at most eight distinct four-character uppercase +alphanumeric IDs, ordered with the active epoch last. Save and install every +listed key before sending it. The last ID must be the provider's current or +pending epoch. Publish a matching profile when changing the active epoch. + +To provision a replacement, include a random `keyRequestId` of 16–128 URL-safe +characters, saved before sending. The response includes +`keyRequest: {id,keyId}` and, on first delivery only, `ticketKey: {keyId,secret}`. +A key secret has 32–256 UTF-8 characters; optional `notBefore` and `retireAfter` +are epoch milliseconds. An idempotent retry cannot mint another key or return +the secret again. If its delivery was lost, use a fresh request ID to provision +a replacement. A provider may retire an unacknowledged, superseded pending key. + +Install the key atomically, then immediately publish its profile and acknowledge +it in heartbeat. Replies include `retirements: [{keyId,retireAfter}]` for older +reported epochs. Repeated replies preserve the original deadlines; they cannot +extend key life. Providers must allow outstanding tokens their defined overlap +window. Reject tokens before activation or after retirement and erase retired +material. Key rotation never extends token expiry. + +### Readiness, lease and schedule + +The reply includes `receivedAt` (ISO8601), `hostProfileRevision`, `activeKeyId`, +`leaseGeneration`, `readiness: {routable,reasons}`, and: + +```text +checkIn: {version:1,afterMillis,nextCheckInAt,leaseExpiresAt,minUpdateIntervalMillis} +``` + +Schedule timestamps are epoch milliseconds. `nextCheckInAt` precedes lease +expiry. `checkInVersion: 1` requests scheduling; while an initial usable profile +is unavailable the provider can omit `checkIn` and use its discovery cadence +and `staleAfter` ISO8601 deadline. A draining/closed host is never routable. + +Readiness is a current provider observation; only the recorded `checkIn` or +`staleAfter` grants a lease. Request replay returns that original grant. Hosts +use monotonic timers, count network time against the interval, and publish +changed activity/status earlier subject to the returned rate limit. Restart +immediately publishes fresh state and discards the prior schedule. Existing +sessions survive a control-plane outage. + +## `outcomes` + +Request: `{events:[{ticketId,stage,occurredAt,reason?}]}` with at most 100 events. +`occurredAt` is ISO8601; `reason` is a bounded code of at most 128 characters. +The ticket ID derives from the authenticated admission carrier, never from an +unauthenticated packet. The provider scopes correlation to the signed instance. +No provider-specific routing decision ID is required. + +Required stages are `ticket.data_channels_open` and `ticket.failed` for observed +transport attempts, plus `ticket.game_joined`/`ticket.game_rejected` when +`gameOutcomes` is `available`. Optional diagnostic stages are `ticket.ice_seen`, +`ticket.ice_connected`, `ticket.dtls_connected` and `ticket.sctp_connected`. +Success and failure describe the observed boundary, not an inferred later stage. + +A successful response acknowledges the whole batch. Repeated observations must +be deduplicated by instance, ticket, stage, occurrence time and reason, including +when a retry uses a new request ID. Queue and persist redacted reports with a +finite bound; the reference client retains at most 1000 pending entries and +flushes at most 100 per tick independently of idle heartbeat timing. Backpressure +must not block native admission or lease renewal. Neither absent reports nor an +unreachable host proves a particular client's outcome. Never send SDP, private +keys, player identity or game payloads. + +## Stateless admission carrier + +### Carry the token in the ICE username + +The client's first STUN USERNAME is `:`, where: + +```text +answerUfrag = "NXS1" + keyId + unpaddedBase64(nonce || ciphertext || tag) +``` + +Use the standard base64 alphabet, including `+` and `/`, which ICE permits. +Do not use base64url. The total ufrag length is at most 256 characters. Before +allocating peer state, reject noncanonical encoding, trailing padding, a wrong +prefix, unknown key epochs, and oversized input. + +AES-256-GCM uses a random 12-byte nonce and a 16-byte tag. Its key is +`HMAC-SHA256(secret, "nxs-stateless-aead-v1" || NUL || audience)`. +The audience is `nxs-stateless-host-v1/`. The additional authenticated +data (AAD) is +`"nxs-stateless-admission-v1" || NUL || ("NXS1"+keyId) || NUL || audience || NUL || clientUfrag`. + +| Plaintext offset | Size | Meaning, unsigned big-endian where numeric | +| --- | --- | --- | +| 0 | 4 | Expiry in epoch seconds, exactly representable in milliseconds | +| 4 | 32 | SHA-256 client certificate fingerprint | +| 36 | 2 | Client SCTP port, 1–65535 | +| 38 | 4 | Client maximum message size, 1–262144 | +| 42 | 16 | Opaque caller-context hash, no account-specific interpretation | +| 58 | 8 | NetherNet network ID, unsigned 64-bit | +| 66 | 1 | Client ICE password length, 22–91 | +| 67 | N | Client ICE password in ICE base64 alphabet | + +The host's local ICE password is the unpadded standard base64 encoding of the +first 24 bytes of +`HMAC-SHA256(secret, "nxs-stateless-ice-v1" || NUL || audience || NUL || answerUfrag)`. +The ticket correlation ID is the first 16 bytes of SHA-256 of the ASCII answer +ufrag, encoded as lowercase hex. + +### Validate the first packet + +A token can be valid for at most 120 seconds. Before assigning the UDP tuple to a +peer or allocating a peer connection, the host checks expiry, field bounds, GCM +authentication, client binding, and STUN MESSAGE-INTEGRITY. The DTLS handshake +MUST then verify the client fingerprint from the token. + +Only a retransmission of the identical token from the same UDP tuple can reuse +a reservation. Reject the same token from another tuple. Also reject a conflicting +admission on an occupied tuple. + +Limit the number of sessions, pending handshakes, used-token records, queued +validation tasks, and retained requests. Duplicate requests for the same pending +attempt share one decision. Preserve enough of the first request to respond after +acceptance: completing admission MUST NOT depend on the client retransmitting. +Release admission capacity only after the connection's resources have been +released. A failed integrity check MUST NOT consume the token, since a copied +token alone does not prove that the sender has its ICE password. + +#### Reference implementation + +The supplied Network implementation uses a 60-second token limit. libjuice +retains the first STUN packet and sends parsed metadata to Java for asynchronous +token validation. libdatachannel verifies STUN integrity before creating the +peer, outside the UDP receive lock. It then processes the retained request after +the application has installed its callbacks. Established transport packets stay +native, and capacity remains reserved until native teardown finishes. + +Other implementations may meet the requirements above using different languages, +threading models, and transport libraries. + +## Optional extensions + +Providers can add optional application metadata without making it part of NXS. +For example, a product could supply an account-claim link. NXS does not define +what claiming an account means or require other providers to implement it. + +`extensions` is an object with at most 16 reverse-DNS namespace keys, such as +`com.example.feature`, and at most 16384 bytes of encoded UTF-8 JSON. Keys use +lowercase domain-style labels and have at most 128 characters. Each value is +`{version:positiveInteger,critical:boolean,data:object}`. + +Pass through or ignore unknown optional extensions; never execute them +automatically. Reject unsupported critical extensions before sending credentials +or publishing readiness. An optional extension cannot change the core protocol rules. +TLS and request signatures still authenticate bodies and operation paths. + +An extension can advertise URLs in `data.operations`. An application can request +one of these operations only after validating the namespace, version, and meaning. +The generic transport still requires the same provider origin and signs the +exact path. + +## Conformance + +Run `node docs/external-signalling/fixtures.mjs` for independent signature and +admission fixtures. The Java tests consume the same schema/fixtures and exercise +an independent provider with no product accounts. Native tests separately cover +local admission and real UDP/ICE/DTLS/SCTP. Report stock-client gameplay separately +from these checks. diff --git a/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/CheckInSchedule.java b/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/CheckInSchedule.java index 560f2b5..4fbbff4 100644 --- a/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/CheckInSchedule.java +++ b/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/CheckInSchedule.java @@ -4,17 +4,17 @@ import java.io.IOException; /** Validates the scheduling contract; policy and idle thresholds belong to the provider. */ -record CheckInSchedule(long afterMillis, long controlPollAfterMillis, long minUpdateIntervalMillis) { +record CheckInSchedule(long afterMillis, long minUpdateIntervalMillis) { static CheckInSchedule parse(JsonObject response) throws IOException { try { JsonObject s = response.getAsJsonObject("checkIn"); if (number(s, "version") != 1) throw new IllegalArgumentException(); - long after = number(s, "afterMillis"), control = number(s, "controlPollAfterMillis"), minimum = number(s, "minUpdateIntervalMillis"); + long after = number(s, "afterMillis"), minimum = number(s, "minUpdateIntervalMillis"); long next = number(s, "nextCheckInAt"), expires = number(s, "leaseExpiresAt"); long received = java.time.Instant.parse(response.get("receivedAt").getAsString()).toEpochMilli(); - if (after < 1000 || after > 86400000 || control < 1000 || control > after || minimum < 1000 || minimum > after + if (after < 1000 || after > 86400000 || minimum < 1000 || minimum > after || next - received != after || expires <= next || expires - next > 300000) throw new IllegalArgumentException(); - return new CheckInSchedule(after, control, minimum); + return new CheckInSchedule(after, minimum); } catch (RuntimeException invalid) { throw new IOException("Invalid provider check-in schedule", invalid); } } private static long number(JsonObject object, String field) { diff --git a/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/ProviderClient.java b/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/ProviderClient.java index 48c27da..56764e8 100644 --- a/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/ProviderClient.java +++ b/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/ProviderClient.java @@ -63,7 +63,9 @@ public static final class ProviderException extends IOException { private JsonObject lastProfile; private long intervalMs = 10000, nextHeartbeat, snapshotClock; private boolean started, closed, scheduledCheckIns; - private long nextControl, nextStatusUpdate, controlIntervalMs = 1000, minUpdateIntervalMs = 1000; + private long nextOutcomes, nextStatusUpdate, minUpdateIntervalMs = 1000, appliedStateRevision; + private String hostState = "serving", installedKeyId; + private JsonObject lastHeartbeat = new JsonObject(); private ServerStatus lastReportedStatus; private Health lastReportedHealth; private final AtomicBoolean closing = new AtomicBoolean(); @@ -86,13 +88,10 @@ public CompletableFuture start() { return submit(() -> { else recoverExisting(); JsonObject registration = state.getAsJsonObject("registration"); if (!registration.get("provider").getAsString().equals(origin)) throw new IOException("Registration audience changed"); - JsonObject activationRequest = new JsonObject(); activationRequest.addProperty("profile", config.profile()); - JsonObject activation = signed("activate", "POST", activationRequest); state.addProperty("protocol", ProviderCrypto.PROTOCOL); state.addProperty("profile", config.profile()); - state.addProperty("generation", activation.get("leaseGeneration").getAsLong()); state.addProperty("sequence", 0); state.remove("cursor"); save(); + state.addProperty("generation", registration.get("leaseGeneration").getAsLong()); state.addProperty("sequence", 0); + state.remove("cursor"); state.remove("pendingAdmissions"); save(); installKeys(); - // Volatile admissions from an earlier profile cannot be restored by a stateless endpoint. - state.remove("pendingAdmissions"); save(); started = true; heartbeat(); timer = executor.scheduleWithFixedDelay(() -> { if (closed) return; @@ -103,11 +102,10 @@ public CompletableFuture start() { return submit(() -> { nextStatusUpdate = nextHeartbeat; diagnostics.accept("provider_status_unavailable: " + safeFailure(e)); } - if (System.nanoTime() >= nextControl) { - nextControl = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(controlIntervalMs); - try { control(); } catch (Exception e) { diagnostics.accept("provider_control_unavailable: " + safeFailure(e)); } + if (System.nanoTime() >= nextOutcomes) { + try { flushEvents(); } + catch (Exception e) { nextOutcomes = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); diagnostics.accept("provider_events_unavailable: " + safeFailure(e)); } } - try { flushEvents(); } catch (Exception e) { diagnostics.accept("provider_events_unavailable: " + safeFailure(e)); } }, 1000, 1000, TimeUnit.MILLISECONDS); return redactedRegistration(); }); } @@ -130,7 +128,6 @@ private void validateDiscovery() throws IOException { if (intervalMs < 1000 || intervalMs > 30000) throw new IOException("Unsupported heartbeat interval"); JsonObject limits = discovery.getAsJsonObject("limits"); if (limits.get("maxBodyBytes").getAsLong() < 1 || limits.get("maxBodyBytes").getAsLong() > 65536 - || limits.get("maxControlPage").getAsLong() < 1 || limits.get("maxControlPage").getAsLong() > 100 || limits.get("clockSkewMs").getAsLong() < 0 || limits.get("clockSkewMs").getAsLong() > 60000) throw new IOException("Unsupported provider limits"); } @@ -141,7 +138,7 @@ private JsonObject recoveryRequest(String registrationId) { } private void recoverExisting() throws Exception { String registrationId = registration("registrationId"); - completeRecovery(unsigned("recover", recoveryRequest(registrationId)), registrationId); + completeRecovery(unsigned("register", recoveryRequest(registrationId)), registrationId); } private void completeRecovery(JsonObject challenge, String registrationId) throws Exception { ProviderContract.require("challenge", challenge); @@ -167,10 +164,10 @@ private void completeRecovery(JsonObject challenge, String registrationId) throw if (!state.getAsJsonObject("registration").get(field).equals(recovered.get(field))) throw new IOException("Recovered instance identity changed"); registrationExtensions = ProtocolExtensions.copy(recovered); recovered.remove("extensions"); recovered.remove("ticketKey"); state.add("registration", recovered); state.addProperty("generation", recovered.get("leaseGeneration").getAsLong()); - if (!state.has("sequence")) state.addProperty("sequence", 0); + state.addProperty("sequence", 0); if (!state.has("ticketKeys")) state.add("ticketKeys", new JsonArray()); state.remove("challenge"); - // Sequence is monotonic within a generation; the previous durable reservation is retained. + // Completion starts a fresh fenced generation; operational sequencing starts at zero. if (pending) { state.add("privateKey", state.remove("pendingPrivateKey")); state.add("publicKeyJwk", state.remove("pendingPublicKeyJwk")); privateKey = key; } save(); } @@ -179,7 +176,7 @@ private void enroll() throws Exception { if (state.has("challenge")) { String registrationId = state.getAsJsonObject("challenge").get("challengeId").getAsString(); JsonObject recoveredChallenge = null; - try { recoveredChallenge = unsigned("recover", recoveryRequest(registrationId)); } + try { recoveredChallenge = unsigned("register", recoveryRequest(registrationId)); } catch (ProviderException e) { if (e.status != 403) throw e; } if (recoveredChallenge != null) { completeRecovery(recoveredChallenge, registrationId); return; } challenge = state.getAsJsonObject("challenge"); @@ -188,7 +185,7 @@ private void enroll() throws Exception { request.addProperty("profile", config.profile()); request.add("publicKeyJwk", state.get("publicKeyJwk")); if (config.label() != null) request.addProperty("label", config.label()); JsonObject authorization = new JsonObject(); authorization.addProperty("scheme", config.authorizationScheme()); request.add("authorization", authorization); if (config.region() != null) { JsonObject p = new JsonObject(); p.addProperty("region", config.region()); p.addProperty("pool", config.pool()); if (!config.tags().isEmpty()) p.add("tags", JSON.toJsonTree(config.tags())); request.add("placement", p); } - challenge = unsigned("challenges", request, config.authorizationToken()); state.add("challenge", challenge); save(); + challenge = unsigned("register", request, config.authorizationToken()); state.add("challenge", challenge); save(); } ProviderContract.require("challenge", challenge); if (!ProviderCrypto.PROTOCOL.equals(challenge.get("protocol").getAsString()) || !ProviderCrypto.SIGNATURE.equals(challenge.get("signature").getAsString()) || !origin.equals(challenge.get("audience").getAsString()) || !ProviderCrypto.thumbprint(state.getAsJsonObject("publicKeyJwk")).equals(challenge.get("thumbprint").getAsString()) || !ProviderCrypto.contextDigest(challenge.getAsJsonObject("context")).equals(challenge.get("contextDigest").getAsString())) throw new IOException("Unbound registration challenge"); @@ -228,27 +225,25 @@ private void validateRegistration(JsonObject registration) throws IOException { if (!expectedTags.equals(actualTags)) throw new IOException("Registration placement tags changed"); } private void installKeys() throws Exception { - if (!state.has("ticketKeys")) state.add("ticketKeys", new JsonArray()); - JsonArray unexpired = new JsonArray(); for (JsonElement e : state.getAsJsonArray("ticketKeys")) if (!e.getAsJsonObject().has("retireAfter") || e.getAsJsonObject().get("retireAfter").getAsLong() > System.currentTimeMillis()) unexpired.add(e); - state.add("ticketKeys", unexpired); save(); - if (state.getAsJsonArray("ticketKeys").isEmpty()) { - JsonObject fresh = signed("ticket-keys", "POST", new JsonObject()); - if (!fresh.has("ticketKey")) throw new IOException("Ticket response was lost; retry fresh provisioning"); - state.getAsJsonArray("ticketKeys").add(fresh.get("ticketKey")); save(); + JsonArray retained = new JsonArray(); + if (state.has("ticketKeys")) for (JsonElement entry : state.getAsJsonArray("ticketKeys")) { + JsonObject key = entry.getAsJsonObject(); + if (!key.has("retireAfter") || key.get("retireAfter").getAsLong() > System.currentTimeMillis()) retained.add(key); + } + if (retained.size() > 8) throw new IOException("Too many admission key epochs"); + state.add("ticketKeys", retained); save(); + if (retained.isEmpty()) { + installedKeyId = null; + if (!state.has("keyRequestId")) { state.addProperty("keyRequestId", UUID.randomUUID().toString()); save(); } + return; } List keys = new ArrayList<>(); - for (JsonElement e : state.getAsJsonArray("ticketKeys")) { JsonObject k = e.getAsJsonObject(); keys.add(new ProviderTransport.TicketKey(k.get("keyId").getAsString(), k.get("secret").getAsString(), k.has("notBefore") ? k.get("notBefore").getAsLong() : 0, k.has("retireAfter") ? k.get("retireAfter").getAsLong() : Long.MAX_VALUE)); } - transport.installTicketKeys(List.copyOf(keys)).toCompletableFuture().get(10, TimeUnit.SECONDS); - JsonObject ack = new JsonObject(); ack.addProperty("keyId", keys.getLast().keyId()); JsonObject acknowledgement = signed("ticket-keys/ack", "POST", ack); - if (acknowledgement.has("retirements")) { - for (JsonElement retired : acknowledgement.getAsJsonArray("retirements")) for (JsonElement stored : state.getAsJsonArray("ticketKeys")) { - JsonObject r = retired.getAsJsonObject(), k = stored.getAsJsonObject(); - if (r.get("keyId").equals(k.get("keyId"))) k.addProperty("retireAfter", Math.min(k.has("retireAfter") ? k.get("retireAfter").getAsLong() : Long.MAX_VALUE, r.get("retireAfter").getAsLong())); - } - save(); List bounded = new ArrayList<>(); - for (JsonElement stored : state.getAsJsonArray("ticketKeys")) { JsonObject k = stored.getAsJsonObject(); long end = k.has("retireAfter") ? k.get("retireAfter").getAsLong() : Long.MAX_VALUE; if (end > System.currentTimeMillis()) bounded.add(new ProviderTransport.TicketKey(k.get("keyId").getAsString(), k.get("secret").getAsString(), k.has("notBefore") ? k.get("notBefore").getAsLong() : 0, end)); } - transport.installTicketKeys(List.copyOf(bounded)).toCompletableFuture().get(10, TimeUnit.SECONDS); + for (JsonElement entry : retained) { JsonObject key = entry.getAsJsonObject(); + keys.add(new ProviderTransport.TicketKey(key.get("keyId").getAsString(), key.get("secret").getAsString(), + key.has("notBefore") ? key.get("notBefore").getAsLong() : 0, key.has("retireAfter") ? key.get("retireAfter").getAsLong() : Long.MAX_VALUE)); } + transport.installTicketKeys(List.copyOf(keys)).toCompletableFuture().get(10, TimeUnit.SECONDS); + installedKeyId = keys.getLast().keyId(); } /** A full immutable snapshot. Callers may update every one of the seven fields. */ public void setServerStatus(ServerStatus status) { explicitStatus.set(Objects.requireNonNull(status)); requestStatusRefresh(); } @@ -266,55 +261,92 @@ private boolean statusChanged() { || !Objects.equals(health.protocolVersion(), lastReportedHealth.protocolVersion()) || !Objects.equals(health.build(), lastReportedHealth.build()); } private void heartbeat() throws Exception { - JsonObject profile = transport.hostProfile().toCompletableFuture().get(10, TimeUnit.SECONDS); - if (profile == null) throw new IOException("Transport profile unavailable"); - boolean supportsSchedule = discovery.getAsJsonObject("limits").has("checkInVersion") - && discovery.getAsJsonObject("limits").get("checkInVersion").getAsInt() == 1 && profile.has("statelessAdmission"); - if (!profile.equals(lastProfile) || !state.has("profilePublishedAt") || (!supportsSchedule && System.currentTimeMillis() - state.get("profilePublishedAt").getAsLong() > 300000)) { - JsonObject published = signed("host-profile", "POST", profile); profileRevision = published.get("revision").getAsString(); lastProfile = profile.deepCopy(); state.addProperty("profilePublishedAt", System.currentTimeMillis()); save(); - } - Health h = healthSupplier.get(); JsonObject body = new JsonObject(); body.addProperty("healthy", h.healthy()); body.addProperty("capacity", h.capacity()); body.addProperty("load", h.load()); body.addProperty("protocolVersion", h.protocolVersion()); body.addProperty("build", h.build()); body.addProperty("hostProfileRevision", profileRevision); - if (config.region() != null) body.addProperty("region", config.region()); - snapshotClock = Math.max(System.currentTimeMillis(), snapshotClock + 1); body.addProperty("clockUnixMillis", snapshotClock); - ServerStatus status = null; - try { status = currentStatus(); if (status != null) body.add("serverStatus", JSON.toJsonTree(status)); } - catch (RuntimeException e) { diagnostics.accept("status_refresh_failed"); /* Omit snapshot; old report timestamp must expire. */ } - if (supportsSchedule) body.addProperty("checkInVersion", 1); - long requestStarted = System.nanoTime(); - JsonObject response = signed("heartbeat", "POST", body); - if (supportsSchedule && response.has("checkIn")) { - CheckInSchedule schedule = CheckInSchedule.parse(response); - scheduledCheckIns = true; controlIntervalMs = schedule.controlPollAfterMillis(); minUpdateIntervalMs = schedule.minUpdateIntervalMillis(); - // Count network time against the granted interval; retries cannot postpone an absolute lease. - long received = java.time.Instant.parse(response.get("receivedAt").getAsString()).toEpochMilli(); - long remaining = Math.min(schedule.afterMillis(), Math.max(0, response.getAsJsonObject("checkIn").get("nextCheckInAt").getAsLong() - Math.max(received, System.currentTimeMillis()))); - nextHeartbeat = Math.min(requestStarted + TimeUnit.MILLISECONDS.toNanos(schedule.afterMillis()), System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(remaining)); - nextControl = Math.min(nextControl, requestStarted + TimeUnit.MILLISECONDS.toNanos(controlIntervalMs)); - } else { - scheduledCheckIns = false; controlIntervalMs = 1000; - nextHeartbeat = requestStarted + TimeUnit.MILLISECONDS.toNanos(intervalMs + ThreadLocalRandom.current().nextLong(Math.max(1, intervalMs / 10))); - nextControl = Math.min(nextControl, System.nanoTime() + TimeUnit.SECONDS.toNanos(1)); - } - nextStatusUpdate = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(scheduledCheckIns ? minUpdateIntervalMs : intervalMs); - lastReportedStatus = status; lastReportedHealth = h; - } - private void control() throws Exception { - JsonObject page = signed("control", "GET", null); - JsonArray commands = page.has("commands") ? page.getAsJsonArray("commands") : new JsonArray(); - if (commands.size() > discovery.getAsJsonObject("limits").get("maxControlPage").getAsInt()) throw new IOException("Control page exceeds limit"); - boolean terminal = true; - for (JsonElement item : commands) { - JsonObject command = item.getAsJsonObject(); String kind = command.has("kind") ? command.get("kind").getAsString() : ""; - if (!Set.of("noop", "drain", "suspend", "revoke").contains(kind)) { - terminal = false; diagnostics.accept("unsupported_control_command"); continue; + // Key delivery and application acknowledgements can need an immediate second exchange. + for (int exchange = 0; exchange < 3; exchange++) { + JsonObject body = new JsonObject(), profile = null; + if (installedKeyId != null && hostState.equals("serving")) { + profile = transport.hostProfile().toCompletableFuture().get(10, TimeUnit.SECONDS); + if (profile == null) throw new IOException("Transport profile unavailable"); + if (!profile.equals(lastProfile)) body.add("hostProfile", profile); + else if (profileRevision != null) body.addProperty("hostProfileRevision", profileRevision); + } else if (profileRevision != null) body.addProperty("hostProfileRevision", profileRevision); + if (installedKeyId != null) { + JsonArray installed = new JsonArray(); + for (JsonElement key : state.getAsJsonArray("ticketKeys")) installed.add(key.getAsJsonObject().get("keyId")); + body.add("installedKeyIds", installed); } - ProviderTransport.ApplyResult result = transport.applyControl(command.deepCopy()).toCompletableFuture().get(10, TimeUnit.SECONDS); - if (result == ProviderTransport.ApplyResult.PENDING) terminal = false; - } - if (terminal && !commands.isEmpty() && page.has("cursor")) { - // Native terminal results are replay-safe; merely staged volatile admissions never reach here. - String cursor = page.get("cursor").getAsString(); JsonObject ack = new JsonObject(); ack.addProperty("cursor", cursor); signed("control/ack", "POST", ack); state.addProperty("cursor", cursor); save(); + if (state.has("keyRequestId")) body.add("keyRequestId", state.get("keyRequestId")); + Health health = healthSupplier.get(); + body.addProperty("healthy", health.healthy() && installedKeyId != null && hostState.equals("serving")); + body.addProperty("capacity", health.capacity()); body.addProperty("load", health.load()); + body.addProperty("protocolVersion", health.protocolVersion()); body.addProperty("build", health.build()); + if (config.region() != null) body.addProperty("region", config.region()); + snapshotClock = Math.max(System.currentTimeMillis(), snapshotClock + 1); + body.addProperty("clockUnixMillis", snapshotClock); body.addProperty("checkInVersion", 1); + body.addProperty("state", hostState); body.addProperty("appliedStateRevision", appliedStateRevision); + body.addProperty("gameOutcomes", transport.supportsGameOutcomes() ? "available" : "unavailable"); + ServerStatus status = null; + try { status = currentStatus(); if (status != null) body.add("serverStatus", JSON.toJsonTree(status)); } + catch (RuntimeException failure) { diagnostics.accept("status_refresh_failed"); } + long requestStarted = System.nanoTime(); + JsonObject response = signed("heartbeat", "POST", body); + ProtocolExtensions.validate(response); + if (body.has("hostProfile")) { + if (!response.has("hostProfileRevision") || response.get("hostProfileRevision").isJsonNull()) throw new IOException("Profile acknowledgement missing"); + profileRevision = response.get("hostProfileRevision").getAsString(); lastProfile = profile.deepCopy(); + state.addProperty("profilePublishedAt", System.currentTimeMillis()); save(); + } + boolean again = false; + if (response.has("ticketKey")) { + JsonObject key = response.remove("ticketKey").getAsJsonObject(); + if (!state.has("keyRequestId") || !response.has("keyRequest") || + !state.get("keyRequestId").equals(response.getAsJsonObject("keyRequest").get("id")) || + !key.get("keyId").equals(response.getAsJsonObject("keyRequest").get("keyId"))) throw new IOException("Unbound admission key response"); + state.getAsJsonArray("ticketKeys").add(key); state.remove("keyRequestId"); save(); + installKeys(); lastProfile = null; again = true; + } else if (state.has("keyRequestId") && response.has("keyRequest") && + state.get("keyRequestId").equals(response.getAsJsonObject("keyRequest").get("id"))) { + // The provider confirms delivery but the one-time response was lost. + state.addProperty("keyRequestId", UUID.randomUUID().toString()); save(); again = true; + } + if (response.has("retirements") && !response.getAsJsonArray("retirements").isEmpty()) { + for (JsonElement retirement : response.getAsJsonArray("retirements")) for (JsonElement stored : state.getAsJsonArray("ticketKeys")) { + JsonObject retired = retirement.getAsJsonObject(), key = stored.getAsJsonObject(); + if (retired.get("keyId").equals(key.get("keyId"))) key.addProperty("retireAfter", Math.min( + key.has("retireAfter") ? key.get("retireAfter").getAsLong() : Long.MAX_VALUE, retired.get("retireAfter").getAsLong())); + } + save(); installKeys(); + } + if (response.has("checkIn")) { + CheckInSchedule schedule = CheckInSchedule.parse(response); + scheduledCheckIns = true; minUpdateIntervalMs = schedule.minUpdateIntervalMillis(); + long received = java.time.Instant.parse(response.get("receivedAt").getAsString()).toEpochMilli(); + long remaining = Math.min(schedule.afterMillis(), Math.max(0, response.getAsJsonObject("checkIn").get("nextCheckInAt").getAsLong() - Math.max(received, System.currentTimeMillis()))); + nextHeartbeat = Math.min(requestStarted + TimeUnit.MILLISECONDS.toNanos(schedule.afterMillis()), System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(remaining)); + } else { + scheduledCheckIns = false; + nextHeartbeat = requestStarted + TimeUnit.MILLISECONDS.toNanos(intervalMs); + } + nextStatusUpdate = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(scheduledCheckIns ? minUpdateIntervalMs : intervalMs); + lastReportedStatus = status; lastReportedHealth = health; lastHeartbeat = response.deepCopy(); + JsonObject desired = response.getAsJsonObject("desiredState"); + if (desired == null || !desired.has("revision") || !desired.has("state")) throw new IOException("Provider state missing"); + long revision = desired.getAsJsonPrimitive("revision").getAsBigDecimal().longValueExact(); String target = desired.get("state").getAsString(); + if (revision < appliedStateRevision || !Set.of("serving", "draining", "closed").contains(target)) throw new IOException("Unsupported provider state"); + if (revision > appliedStateRevision) { + ProviderTransport.ApplyResult applied = target.equals("serving") ? ProviderTransport.ApplyResult.APPLIED + : transport.applyState(target).toCompletableFuture().get(10, TimeUnit.SECONDS); + if (applied == ProviderTransport.ApplyResult.APPLIED) { + appliedStateRevision = revision; + if (!target.equals("serving")) { hostState = target; again = true; } + } else { + diagnostics.accept("provider_state_not_applied"); + nextHeartbeat = Math.min(nextHeartbeat, System.nanoTime() + TimeUnit.SECONDS.toNanos(1)); + } + } + if (!again) return; } + nextHeartbeat = System.nanoTime() + TimeUnit.SECONDS.toNanos(1); } private void flushEvents() throws Exception { if (!state.has("pendingEvents")) state.add("pendingEvents", new JsonArray()); @@ -324,23 +356,19 @@ private void flushEvents() throws Exception { for (JsonObject event : fresh) { // Persist only the existing redacted telemetry fields, never native SDP or secret extensions. JsonObject safe = new JsonObject(); - for (String field : List.of("stage", "type", "ticketId", "decisionId", "occurredAt", "reason")) if (event.has(field)) safe.add(field, event.get(field)); - if ((!safe.has("stage") && !safe.has("type")) || !safe.has("occurredAt")) throw new IOException("Malformed transport event"); + for (String field : List.of("stage", "ticketId", "occurredAt", "reason")) if (event.has(field)) safe.add(field, event.get(field)); + if (!safe.has("stage") || !safe.has("ticketId") || !safe.has("occurredAt")) throw new IOException("Malformed transport event"); pending.add(safe); } if (pending.isEmpty()) return; save(); - for (String operation : List.of("ticket-events", "events")) { - JsonArray batch = new JsonArray(); - for (JsonElement e : pending) if (e.getAsJsonObject().has(operation.equals("events") ? "type" : "stage") && batch.size() < 100) batch.add(e); - if (batch.isEmpty()) continue; - JsonObject body = new JsonObject(); body.add("events", batch); signed(operation, "POST", body); - for (JsonElement sent : batch) pending.remove(sent); save(); - } + JsonArray batch = new JsonArray(); + for (JsonElement event : pending) if (batch.size() < 100) batch.add(event); + JsonObject body = new JsonObject(); body.add("events", batch); signed("outcomes", "POST", body); + for (JsonElement sent : batch) pending.remove(sent); save(); } - public CompletableFuture readiness() { return submit(() -> { - JsonObject response = signed("readiness", "GET", null); ProtocolExtensions.validate(response); return response; - }); } + /** Refresh through the ordinary heartbeat and return its readiness observation. */ + public CompletableFuture readiness() { return submit(() -> { heartbeat(); return lastHeartbeat.deepCopy(); }); } /** Opaque optional extension metadata; the application decides what it means. */ public CompletableFuture extensions() { return submit(() -> registrationExtensions.deepCopy()); } /** Explicit application request to an advertised extension operation, never automatic execution. */ @@ -362,8 +390,9 @@ public CompletableFuture deregister() { return submit(() -> { signed("deregister", "POST", new JsonObject()); transport.drain().toCompletableFuture().get(10, TimeUnit.SECONDS); started = false; return null; }); } public CompletableFuture rotateTicketKey() { return submit(() -> { - JsonObject result = signed("ticket-keys", "POST", new JsonObject()); if (!result.has("ticketKey")) throw new IOException("Fresh ticket provisioning required"); - state.getAsJsonArray("ticketKeys").add(result.get("ticketKey")); save(); installKeys(); lastProfile = null; heartbeat(); return redactedRegistration(); + installKeys(); + if (state.getAsJsonArray("ticketKeys").size() >= 8) throw new IOException("Wait for retiring admission epochs before rotating again"); + state.addProperty("keyRequestId", UUID.randomUUID().toString()); save(); heartbeat(); return redactedRegistration(); }); } public CompletableFuture rotateMachineKey() { return submit(() -> { KeyPair replacement = ProviderCrypto.generate(); JsonObject jwk = ProviderCrypto.publicJwk(replacement.getPublic()); @@ -374,15 +403,18 @@ public CompletableFuture rotateMachineKey() { return submit(() -> { state.add("privateKey", state.remove("pendingPrivateKey")); state.add("publicKeyJwk", state.remove("pendingPublicKeyJwk")); state.getAsJsonObject("registration").addProperty("keyId", result.get("keyId").getAsString()); save(); privateKey = replacement.getPrivate(); JsonObject retire = new JsonObject(); retire.addProperty("keyId", oldKey); signed("retire", "POST", retire); return result; }); } - public CompletableFuture drain() { return submit(() -> { signed("drain", "POST", new JsonObject()); transport.drain().toCompletableFuture().get(10, TimeUnit.SECONDS); started = false; return null; }); } + public CompletableFuture drain() { return submit(() -> { drainAndReport(); started = false; return null; }); } + private void drainAndReport() throws Exception { + if (!hostState.equals("closed")) { transport.drain().toCompletableFuture().get(10, TimeUnit.SECONDS); hostState = "draining"; } + heartbeat(); + } private JsonObject unsigned(String op, JsonObject body) throws Exception { return unsigned(op, body, null); } private JsonObject unsigned(String op, JsonObject body, String bearerToken) throws Exception { return exchange(operation(op), "POST", JSON.toJson(body), false, null, bearerToken); } private JsonObject signed(String op, String method, JsonObject body) throws Exception { return signed(op, method, body, UUID.randomUUID().toString()); } private JsonObject signed(String op, String method, JsonObject body, String intent) throws Exception { long sequence = state.has("sequence") ? state.get("sequence").getAsLong() + 1 : 1; state.addProperty("sequence", sequence); save(); URI uri = operation(op); - if (op.equals("control") && state.has("cursor")) uri = URI.create(uri + "?cursor=" + java.net.URLEncoder.encode(state.get("cursor").getAsString(), java.nio.charset.StandardCharsets.UTF_8)); - return exchange(uri, method, body == null ? null : JSON.toJson(body), true, intent, null); + return exchange(uri, method, body == null ? null : JSON.toJson(body), true, intent, null, op.equals("outcomes") ? 3 : 15, op.equals("outcomes") ? 1 : 3); } private URI operation(String op) throws IOException { if (!discovery.getAsJsonObject("operations").has(op)) throw new IOException("Missing provider operation: " + op); return trusted(URI.create(discovery.getAsJsonObject("operations").get(op).getAsString())); } private URI trusted(URI uri) throws IOException { @@ -390,9 +422,12 @@ private URI trusted(URI uri) throws IOException { if (!ProviderCrypto.origin(authority).equals(origin) || uri.getUserInfo() != null || uri.getFragment() != null) throw new IOException("Untrusted provider operation"); return uri; } private JsonObject exchange(URI uri, String method, String body, boolean signed, String intent, String bearerToken) throws Exception { + return exchange(uri, method, body, signed, intent, bearerToken, 15, 3); + } + private JsonObject exchange(URI uri, String method, String body, boolean signed, String intent, String bearerToken, int timeoutSeconds, int attempts) throws Exception { trusted(uri); String raw = body == null ? "" : body; - for (int attempt = 0; attempt < 3; attempt++) { - HttpRequest.Builder b = HttpRequest.newBuilder(uri).timeout(Duration.ofSeconds(15)).header("accept", "application/json").method(method, body == null ? HttpRequest.BodyPublishers.noBody() : HttpRequest.BodyPublishers.ofString(body)); + for (int attempt = 0; attempt < attempts; attempt++) { + HttpRequest.Builder b = HttpRequest.newBuilder(uri).timeout(Duration.ofSeconds(timeoutSeconds)).header("accept", "application/json").method(method, body == null ? HttpRequest.BodyPublishers.noBody() : HttpRequest.BodyPublishers.ofString(body)); if (body != null) b.header("content-type", "application/json"); if (bearerToken != null) b.header("authorization", "Bearer " + bearerToken); if (signed) { @@ -402,15 +437,15 @@ private JsonObject exchange(URI uri, String method, String body, boolean signed, } HttpResponse response; var responseFuture = http.sendAsync(b.build(), info -> new LimitedBodySubscriber(65536)); - try { response = responseFuture.get(20, TimeUnit.SECONDS); } + try { response = responseFuture.get(timeoutSeconds + 1, TimeUnit.SECONDS); } catch (ExecutionException | TimeoutException failure) { responseFuture.cancel(true); - if (attempt == 2) throw new IOException("Provider transport unavailable", failure); + if (attempt == attempts - 1) throw new IOException("Provider transport unavailable", failure); Thread.sleep((250L << attempt) + ThreadLocalRandom.current().nextLong(100)); continue; } String text = new String(response.body(), java.nio.charset.StandardCharsets.UTF_8); int status = response.statusCode(); - if ((status == 429 || status == 503 || status == 502 || status == 504) && attempt < 2) { long delay = 250L << attempt; + if ((status == 429 || status == 503 || status == 502 || status == 504) && attempt < attempts - 1) { long delay = 250L << attempt; try { delay = Math.max(delay, Long.parseLong(response.headers().firstValue("retry-after").orElse("0")) * 1000); } catch (NumberFormatException ignored) { } if (delay > 10000) throw new ProviderException(status, "retry_later"); Thread.sleep(delay + ThreadLocalRandom.current().nextLong(100)); continue; } @@ -431,7 +466,7 @@ private CompletableFuture submit(Callable fn) { } public CompletionStage stop() { if (!closing.compareAndSet(false, true)) return stopped; - executor.execute(() -> { try { if (started) { signed("drain", "POST", new JsonObject()); transport.drain().toCompletableFuture().get(10, TimeUnit.SECONDS); } } catch (Exception e) { diagnostics.accept("provider_drain_unavailable"); } + executor.execute(() -> { try { if (started) { drainAndReport(); flushEvents(); } } catch (Exception e) { diagnostics.accept("provider_drain_unavailable"); } finally { closed = true; started = false; if (timer != null) timer.cancel(false); try { transport.close().toCompletableFuture().get(10, TimeUnit.SECONDS); } diff --git a/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/ProviderTransport.java b/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/ProviderTransport.java index 625ab3f..dcffb46 100644 --- a/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/ProviderTransport.java +++ b/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/ProviderTransport.java @@ -11,9 +11,11 @@ enum ApplyResult { PENDING, APPLIED, REJECTED } CompletionStage hostProfile(); /** Atomic snapshot; completion means every supplied key is persisted and usable. */ CompletionStage installTicketKeys(List keys); - /** Existing complete AgentControlCommand envelope. PENDING holds whole-page acknowledgement. */ - CompletionStage applyControl(JsonObject command); - /** Bounded events using existing ticket.* and separate authenticated game_joined semantics. */ + /** Apply serving/draining/closed background state before acknowledging its revision. */ + CompletionStage applyState(String state); + /** Whether this integration can observe the application join/rejection boundary. */ + default boolean supportsGameOutcomes() { return false; } + /** Bounded ticket-correlated transport and application observations. */ List pollEvents(); CompletionStage drain(); CompletionStage close(); diff --git a/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/admission/NativeProviderTransport.java b/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/admission/NativeProviderTransport.java index 6ff1eae..dfc8832 100644 --- a/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/admission/NativeProviderTransport.java +++ b/external-signalling/src/main/java/org/cloudburstmc/netty/signalling/admission/NativeProviderTransport.java @@ -105,14 +105,12 @@ private static List checkedEndpoints(List return CompletableFuture.completedFuture(null); } catch (Exception invalid) { return CompletableFuture.failedFuture(invalid); } } - @Override public CompletionStage applyControl(JsonObject command) { - if (command == null || !command.has("kind") || !command.get("kind").isJsonPrimitive() || !command.getAsJsonPrimitive("kind").isString()) - return CompletableFuture.completedFuture(ApplyResult.REJECTED); - return switch (command.get("kind").getAsString()) { - case "noop" -> CompletableFuture.completedFuture(ApplyResult.APPLIED); - case "drain" -> drain().thenApply(ignored -> ApplyResult.APPLIED); - case "suspend", "revoke" -> close().thenApply(ignored -> ApplyResult.APPLIED); - // Native admission never stages a client from control. Unsupported lifecycle changes are explicit rejections. + @Override public CompletionStage applyState(String state) { + if (state == null) return CompletableFuture.completedFuture(ApplyResult.REJECTED); + return switch (state) { + case "serving" -> CompletableFuture.completedFuture(ApplyResult.APPLIED); + case "draining" -> drain().thenApply(ignored -> ApplyResult.APPLIED); + case "closed" -> close().thenApply(ignored -> ApplyResult.APPLIED); default -> CompletableFuture.completedFuture(ApplyResult.REJECTED); }; } diff --git a/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/CheckInScheduleTest.java b/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/CheckInScheduleTest.java index 7c5357b..314c198 100644 --- a/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/CheckInScheduleTest.java +++ b/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/CheckInScheduleTest.java @@ -11,12 +11,12 @@ private JsonObject response(long delay) { response.addProperty("receivedAt", java.time.Instant.ofEpochMilli(now).toString()); schedule.addProperty("version", 1); schedule.addProperty("afterMillis", delay); schedule.addProperty("nextCheckInAt", now + delay); schedule.addProperty("leaseExpiresAt", now + delay + 30000); - schedule.addProperty("controlPollAfterMillis", delay); schedule.addProperty("minUpdateIntervalMillis", 1000); + schedule.addProperty("minUpdateIntervalMillis", 1000); response.add("checkIn", schedule); return response; } @Test void acceptsChangedPolicyWithoutHardCodedIdleThresholds() throws Exception { assertEquals(900000, CheckInSchedule.parse(response(900000)).afterMillis()); - assertEquals(3600000, CheckInSchedule.parse(response(3600000)).controlPollAfterMillis()); + assertEquals(3600000, CheckInSchedule.parse(response(3600000)).afterMillis()); assertEquals(45000, CheckInSchedule.parse(response(45000)).afterMillis()); } @Test void rejectsUnboundedFractionalAndInconsistentSchedules() { diff --git a/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/IndependentProviderStub.java b/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/IndependentProviderStub.java index e511c7b..5ddf96f 100644 --- a/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/IndependentProviderStub.java +++ b/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/IndependentProviderStub.java @@ -15,6 +15,8 @@ public final class IndependentProviderStub implements AutoCloseable { final Map challenges = new HashMap<>(), keys = new HashMap<>(), placements = new HashMap<>(); JsonObject registration; volatile JsonObject lastHeartbeat; volatile int failHeartbeats; + volatile boolean failOutcomes; + volatile int outcomeAttempts; volatile boolean loseCompletionResponse; volatile long checkInMillis; volatile int controlPolls; @@ -26,7 +28,11 @@ public final class IndependentProviderStub implements AutoCloseable { volatile JsonObject extensionMetadata; volatile int extensionRequests, keyAcknowledgements; boolean draining; - volatile JsonArray commands = new JsonArray(); + volatile String desiredState = "serving"; + volatile long desiredRevision = 1, appliedRevision; + String keyRequestId; JsonObject requestedKey; + int epoch = 1, profileRevision; + final List operationsSeen = new java.util.concurrent.CopyOnWriteArrayList<>(); public IndependentProviderStub() throws IOException { server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); origin = "http://127.0.0.1:" + server.getAddress().getPort(); server.createContext("/", this::handle); server.start(); @@ -44,14 +50,15 @@ private JsonObject dispatch(HttpExchange e) throws Exception { if (path.equals("/.well-known/nethernet-external-signalling")) { JsonObject d = new JsonObject(); d.addProperty("provider", origin); d.addProperty("controlOrigin", origin); d.add("protocols", strings(ProviderCrypto.PROTOCOL)); d.add("signatures", strings(ProviderCrypto.SIGNATURE)); d.add("modes", strings("new-service", "attach-instance")); d.add("profiles", strings("nxs-admission-v1")); - JsonObject operations = new JsonObject(); for (String op : List.of("challenges", "complete", "recover", "activate", "heartbeat", "host-profile", "readiness", "control", "control/ack", "drain", "rotate", "retire", "ticket-keys", "ticket-keys/ack", "ticket-events", "events", "deregister")) operations.addProperty(op, origin + "/example/" + op); + JsonObject operations = new JsonObject(); for (String op : List.of("register", "complete", "heartbeat", "outcomes", "rotate", "retire", "deregister")) operations.addProperty(op, origin + "/example/" + op); if (extensionMetadata != null) d.add("extensions", extensionMetadata.deepCopy()); - d.add("operations", operations); JsonObject limits = new JsonObject(); limits.addProperty("heartbeatIntervalMs", 1000); if (checkInMillis > 0) limits.addProperty("checkInVersion", 1); limits.addProperty("maxControlPage", 100); limits.addProperty("leaseMs", 30000); limits.addProperty("maxBodyBytes", 65536); limits.addProperty("clockSkewMs", 60000); d.add("limits", limits); + d.add("operations", operations); JsonObject limits = new JsonObject(); limits.addProperty("heartbeatIntervalMs", 1000); if (checkInMillis > 0) limits.addProperty("checkInVersion", 1); limits.addProperty("leaseMs", 30000); limits.addProperty("maxBodyBytes", 65536); limits.addProperty("clockSkewMs", 60000); d.add("limits", limits); JsonObject authorization = new JsonObject(); authorization.addProperty("header", "Authorization"); JsonArray schemes = new JsonArray(); schemes.add(authorizationScheme("anonymous-proof-of-work", "new-service")); schemes.add(authorizationScheme("bearer-token", "new-service", "attach-instance")); authorization.add("schemes", schemes); d.add("authorization", authorization); return d; } - if (path.equals("/example/challenges") || path.equals("/example/recover")) { - boolean recovery = path.endsWith("recover"); + operationsSeen.add(path); + if (path.equals("/example/register")) { + boolean recovery = body.has("registrationId"); if (!ProviderCrypto.PROTOCOL.equals(body.get("protocol").getAsString()) || !"nxs-admission-v1".equals(body.get("profile").getAsString())) throw new Failure(400, "unsupported_profile"); if (recovery && (registration == null || !registration.get("registrationId").equals(body.get("registrationId")))) throw new Failure(403, "recovery_unavailable"); JsonObject key = recovery ? keys.get(registration.get("keyId").getAsString()) : body.getAsJsonObject("publicKeyJwk"); @@ -75,38 +82,48 @@ private JsonObject dispatch(HttpExchange e) throws Exception { if (c == null) throw new Failure(409, "challenge_consumed"); String proof = ProviderCrypto.proof(c, body.get("proofNonce").getAsString(), body.get("idempotencyKey").getAsString()); if (!ProviderCrypto.verify(keys.get(id), body.get("signature").getAsString(), proof) || !ProviderCrypto.meetsDifficulty(ProviderCrypto.digest(proof), c.getAsJsonObject("pow").get("difficulty").getAsInt())) throw new Failure(401, "proof_invalid"); - challenges.remove(id); + challenges.remove(id); generation++; sequence = 0; draining = false; appliedRevision = 0; profileRevision = 0; if (c.getAsJsonObject("context").get("mode").getAsString().equals("recover")) { JsonObject r = registration.deepCopy(); r.remove("ticketKey"); r.addProperty("leaseGeneration", generation); return r; } if (registration != null) throw new Failure(409, "already_registered"); registrations++; - registration = new JsonObject(); registration.addProperty("protocol", ProviderCrypto.PROTOCOL); registration.addProperty("provider", origin); registration.addProperty("registrationId", id); registration.addProperty("instanceId", "example-machine-1"); registration.addProperty("serviceId", "example-service-1"); registration.addProperty("keyId", "example-key-1"); registration.addProperty("publicAddress", "https://play.example.invalid"); registration.addProperty("profile", "nxs-admission-v1"); registration.addProperty("leaseGeneration", 0); registration.addProperty("leaseDeadline", 0); registration.addProperty("heartbeatIntervalMs", 1000); JsonObject ready = new JsonObject(); ready.addProperty("routable", false); ready.add("reasons", new JsonArray()); registration.add("readiness", ready); JsonObject place = placements.getOrDefault(id, new JsonObject()).deepCopy(); if (!place.has("region")) place.addProperty("region", ""); if (!place.has("pool")) place.addProperty("pool", ""); registration.add("placement", place); registration.add("ticketKey", ticket()); if (extensionMetadata != null) registration.add("extensions", extensionMetadata.deepCopy()); keys.put("example-key-1", keys.get(id)); if (loseCompletionResponse) { loseCompletionResponse = false; e.close(); throw new Failure(503, "completion_response_lost"); } return registration.deepCopy(); + registration = new JsonObject(); registration.addProperty("protocol", ProviderCrypto.PROTOCOL); registration.addProperty("provider", origin); registration.addProperty("registrationId", id); registration.addProperty("instanceId", "example-machine-1"); registration.addProperty("serviceId", "example-service-1"); registration.addProperty("keyId", "example-key-1"); registration.addProperty("publicAddress", "https://play.example.invalid"); registration.addProperty("profile", "nxs-admission-v1"); registration.addProperty("leaseGeneration", generation); registration.addProperty("leaseDeadline", System.currentTimeMillis() + 30000); registration.addProperty("heartbeatIntervalMs", 1000); JsonObject ready = new JsonObject(); ready.addProperty("routable", false); ready.add("reasons", new JsonArray()); registration.add("readiness", ready); JsonObject place = placements.getOrDefault(id, new JsonObject()).deepCopy(); if (!place.has("region")) place.addProperty("region", ""); if (!place.has("pool")) place.addProperty("pool", ""); registration.add("placement", place); registration.add("ticketKey", ticket()); if (extensionMetadata != null) registration.add("extensions", extensionMetadata.deepCopy()); keys.put("example-key-1", keys.get(id)); if (loseCompletionResponse) { loseCompletionResponse = false; e.close(); throw new Failure(503, "completion_response_lost"); } return registration.deepCopy(); } if (path.equals("/example/heartbeat") && failHeartbeats-- > 0) throw new Failure(503, "fixture_transient"); authenticate(e, raw); JsonObject ok = new JsonObject(); ok.addProperty("accepted", true); switch (path) { - case "/example/activate" -> { if (!"nxs-admission-v1".equals(body.get("profile").getAsString())) throw new Failure(400, "unsupported_profile"); generation++; sequence = 0; draining = false; ok.addProperty("leaseGeneration", generation); ok.addProperty("leaseDeadline", System.currentTimeMillis() + 30000); } - case "/example/host-profile" -> { - if (keyAcknowledgements == 0 || !"nethernet.stateless-admission.v1".equals(body.getAsJsonObject("statelessAdmission").get("capability").getAsString()) - || !body.get("dtlsFingerprint").getAsString().matches("sha-256 [0-9A-F]{2}(?::[0-9A-F]{2}){31}")) throw new Failure(400, "invalid_host_profile"); - ok.addProperty("revision", "example-profile-revision"); - } - case "/example/heartbeat" -> { if (draining) throw new Failure(403, "draining"); lastHeartbeat = body; heartbeats++; + case "/example/heartbeat" -> { + lastHeartbeat = body; heartbeats++; + if (body.has("installedKeyIds")) keyAcknowledgements++; + if (body.has("hostProfile")) { + JsonObject profile = body.getAsJsonObject("hostProfile"); + if (keyAcknowledgements == 0 || !"nethernet.stateless-admission.v1".equals(profile.getAsJsonObject("statelessAdmission").get("capability").getAsString()) + || !profile.get("dtlsFingerprint").getAsString().matches("sha-256 [0-9A-F]{2}(?::[0-9A-F]{2}){31}")) throw new Failure(400, "invalid_host_profile"); + profileRevision++; + } + if (body.has("keyRequestId")) { + String wanted = body.get("keyRequestId").getAsString(); + if (!wanted.equals(keyRequestId)) { keyRequestId = wanted; requestedKey = ticket(); requestedKey.addProperty("keyId", String.format("T%03d", ++epoch)); ok.add("ticketKey", requestedKey.deepCopy()); } + JsonObject request = new JsonObject(); request.addProperty("id", keyRequestId); request.add("keyId", requestedKey.get("keyId")); ok.add("keyRequest", request); + } + draining = !body.get("state").getAsString().equals("serving"); + long applied = body.get("appliedStateRevision").getAsLong(); + if (applied > appliedRevision) { appliedRevision = applied; acknowledgements++; } + JsonObject desired = new JsonObject(); desired.addProperty("revision", desiredRevision); desired.addProperty("state", desiredState); ok.add("desiredState", desired); + ok.addProperty("hostProfileRevision", "example-profile-" + profileRevision); + ok.addProperty("routable", profileRevision > 0 && keyAcknowledgements > 0 && !draining); + JsonObject ready = new JsonObject(); ready.addProperty("routable", ok.get("routable").getAsBoolean()); ready.add("reasons", new JsonArray()); ok.add("readiness", ready); + if (extensionMetadata != null) ok.add("extensions", extensionMetadata.deepCopy()); if (checkInMillis > 0 && body.has("checkInVersion")) { long now = System.currentTimeMillis(); JsonObject schedule = new JsonObject(); schedule.addProperty("version", 1); schedule.addProperty("afterMillis", checkInMillis); schedule.addProperty("nextCheckInAt", now + checkInMillis); schedule.addProperty("leaseExpiresAt", now + checkInMillis + 30000); - schedule.addProperty("minUpdateIntervalMillis", 1000); schedule.addProperty("controlPollAfterMillis", checkInMillis); + schedule.addProperty("minUpdateIntervalMillis", 1000); ok.add("checkIn", schedule); ok.addProperty("receivedAt", java.time.Instant.ofEpochMilli(now).toString()); - } } - case "/example/control" -> { controlPolls++; ok.add("commands", commands.deepCopy()); ok.addProperty("cursor", "example-cursor"); ok.addProperty("serverTime", java.time.Instant.now().toString()); } - case "/example/control/ack" -> { acknowledgements++; commands = new JsonArray(); } - case "/example/readiness" -> { ok.addProperty("routable", heartbeats > 0 && !draining); if (extensionMetadata != null) ok.add("extensions", extensionMetadata.deepCopy()); } + } + } case "/example/extension" -> { extensionRequests++; } case "/example/deregister" -> { draining = true; } - case "/example/drain" -> draining = true; - case "/example/ticket-keys" -> ok.add("ticketKey", ticket()); - case "/example/ticket-keys/ack" -> { keyAcknowledgements++; } - case "/example/ticket-events", "/example/events" -> { for (JsonElement event : body.getAsJsonArray("events")) events.add(event.getAsJsonObject()); } + case "/example/outcomes" -> { outcomeAttempts++; if (failOutcomes) throw new Failure(503, "fixture_outcome_unavailable"); for (JsonElement event : body.getAsJsonArray("events")) events.add(event.getAsJsonObject()); } case "/example/rotate" -> { String old = e.getRequestHeaders().getFirst("nxs-key-id"), intent = e.getRequestHeaders().getFirst("idempotency-key"); JsonObject key = body.getAsJsonObject("publicKeyJwk"); if (!ProviderCrypto.verify(key, body.get("proof").getAsString(), ProviderCrypto.array(ProviderCrypto.PROTOCOL, "rotate", origin, "example-machine-1", old, ProviderCrypto.thumbprint(key), generation, intent))) throw new Failure(401, "replacement_proof_invalid"); diff --git a/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/ProviderBench.java b/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/ProviderBench.java index 7d5285e..6310ee4 100644 --- a/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/ProviderBench.java +++ b/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/ProviderBench.java @@ -21,7 +21,7 @@ public CompletionStage hostProfile() { JsonObject p = new JsonObject(); p.addProperty("credentialKeyId", keyId); p.addProperty("dtlsFingerprint", "sha-256 " + String.join(":", Collections.nCopies(32, "11"))); p.addProperty("sctpPort", 5000); p.addProperty("maxMessageSize", 262144); JsonObject c = new JsonObject(); c.addProperty("address", "127.0.0.1"); c.addProperty("port", 19133); c.addProperty("foundation", "fixture"); c.addProperty("component", 1); c.addProperty("priority", 100); c.addProperty("protocol", "udp"); c.addProperty("type", "host"); JsonArray candidates = new JsonArray(); candidates.add(c); p.add("candidates", candidates); JsonObject capability = new JsonObject(); capability.addProperty("capability", "nethernet.stateless-admission.v1"); capability.addProperty("incarnation", incarnation); p.add("statelessAdmission", capability); return CompletableFuture.completedFuture(p); } - public CompletionStage applyControl(JsonObject c) { return CompletableFuture.completedFuture(ApplyResult.REJECTED); } + public CompletionStage applyState(String state) { return CompletableFuture.completedFuture(ApplyResult.REJECTED); } public List pollEvents() { return List.of(); } public CompletionStage drain() { return CompletableFuture.completedFuture(null); } public CompletionStage close() { return CompletableFuture.completedFuture(null); } diff --git a/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/ProviderClientTest.java b/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/ProviderClientTest.java index 007d90d..0016552 100644 --- a/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/ProviderClientTest.java +++ b/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/ProviderClientTest.java @@ -55,15 +55,15 @@ class ProviderClientTest { () -> new ProviderClient.Health(true, 40, players.get() / 40.0, "nethernet", "fixture"), message -> {}); try { client.start().get(20, TimeUnit.SECONDS); - eventually(() -> stub.controlPolls == 1); + assertEquals(0, stub.controlPolls); Thread.sleep(2200); - assertEquals(1, stub.heartbeats); assertEquals(1, stub.controlPolls); + assertEquals(1, stub.heartbeats); assertEquals(0, stub.controlPolls); for (int i = 0; i < 100; i++) client.requestStatusRefresh(); Thread.sleep(1200); assertEquals(1, stub.heartbeats, "Unchanged local refreshes must not send requests"); stub.checkInMillis = 1000; players.set(1); client.requestStatusRefresh(); eventually(() -> stub.lastHeartbeat.getAsJsonObject("serverStatus").get("players").getAsInt() == 1); int busyBefore = stub.heartbeats; - eventually(() -> stub.heartbeats > busyBefore && stub.controlPolls > 1); + eventually(() -> stub.heartbeats > busyBefore && stub.controlPolls == 0); stub.checkInMillis = 3600000; players.set(0); client.requestStatusRefresh(); eventually(() -> stub.lastHeartbeat.getAsJsonObject("serverStatus").get("players").getAsInt() == 0); Thread.sleep(2200); int before = stub.heartbeats; int polls = stub.controlPolls; @@ -89,16 +89,16 @@ static final class FakeTransport implements ProviderTransport { final CompletableFuture closed = new CompletableFuture<>(); final java.util.Queue events = new java.util.concurrent.ConcurrentLinkedQueue<>(); volatile int installed, applied, admissions, drains; - boolean stateless = true; + boolean stateless = true; String ticketKeyId = "T001"; volatile ApplyResult result = ApplyResult.APPLIED; - public CompletionStage hostProfile() { JsonObject p = new JsonObject(); p.addProperty("credentialKeyId", "T001"); p.addProperty("dtlsFingerprint", "sha-256 " + String.join(":", Collections.nCopies(32, "11"))); p.addProperty("sctpPort", 5000); p.addProperty("maxMessageSize", 262144); + public CompletionStage hostProfile() { JsonObject p = new JsonObject(); p.addProperty("credentialKeyId", ticketKeyId); p.addProperty("dtlsFingerprint", "sha-256 " + String.join(":", Collections.nCopies(32, "11"))); p.addProperty("sctpPort", 5000); p.addProperty("maxMessageSize", 262144); JsonObject c = new JsonObject(); c.addProperty("foundation", "fixture"); c.addProperty("component", 1); c.addProperty("protocol", "udp"); c.addProperty("priority", 100); c.addProperty("address", "127.0.0.1"); c.addProperty("port", 19133); c.addProperty("type", "host"); JsonArray candidates = new JsonArray(); candidates.add(c); p.add("candidates", candidates); if (stateless) { JsonObject cap = new JsonObject(); cap.addProperty("capability", "nethernet.stateless-admission.v1"); cap.addProperty("incarnation", "0123456789abcdef0123456789abcdef"); p.add("statelessAdmission", cap); } return CompletableFuture.completedFuture(p); } - public CompletionStage installTicketKeys(List keys) { installed = keys.size(); return CompletableFuture.completedFuture(null); } - public CompletionStage applyControl(JsonObject c) { - applied++; String kind = c.get("kind").getAsString(); + public CompletionStage installTicketKeys(List keys) { installed = keys.size(); ticketKeyId = keys.getLast().keyId(); return CompletableFuture.completedFuture(null); } + public CompletionStage applyState(String state) { + applied++; String kind = state; if (kind.equals("join-admission")) admissions++; - if (kind.equals("drain")) drains++; + if (kind.equals("draining")) drains++; return CompletableFuture.completedFuture(kind.equals("join-admission") ? result : ApplyResult.APPLIED); } public List pollEvents() { List batch = new ArrayList<>(); for (JsonObject event; (event = events.poll()) != null;) batch.add(event); return batch; } @@ -128,7 +128,7 @@ private static void eventually(java.util.function.BooleanSupplier condition) thr while (!condition.getAsBoolean() && System.nanoTime() < deadline) Thread.sleep(40); assertTrue(condition.getAsBoolean(), "Timed out waiting for provider lifecycle"); } - @Test void failedRefreshRetriesAndRejectsPerJoinControlWithoutBlockingDrain(@TempDir Path path) throws Exception { + @Test void failedRefreshRetriesAndRejectsUnknownStateWithoutAcknowledgingIt(@TempDir Path path) throws Exception { try (IndependentProviderStub stub = new IndependentProviderStub()) { AtomicInteger players = new AtomicInteger(2); FakeTransport host = new FakeTransport(); var config = new ProviderClient.Configuration(URI.create(stub.origin), "nxs-admission-v1", "Example"); @@ -143,19 +143,39 @@ private static void eventually(java.util.function.BooleanSupplier condition) thr for (int i = 0; i < 500; i++) client.requestStatusRefresh(); eventually(() -> stub.lastHeartbeat.has("serverStatus") && stub.lastHeartbeat.getAsJsonObject("serverStatus").get("players").getAsInt() == 5); assertTrue(stub.heartbeats - before <= 2, "Burst must coalesce within heartbeat cadence"); - JsonObject join = new JsonObject(); join.addProperty("kind", "join-admission"); - JsonObject unknown = new JsonObject(); unknown.addProperty("kind", "future-command"); - JsonObject drain = new JsonObject(); drain.addProperty("kind", "drain"); - JsonArray commands = new JsonArray(); commands.add(join); commands.add(unknown); commands.add(drain); stub.commands = commands; - eventually(() -> host.drains > 0); - assertEquals(0, host.admissions, "NXS never stages a join from provider control"); - assertEquals(0, stub.acknowledgements, "Unsupported control must not be silently acknowledged"); - JsonArray known = new JsonArray(); known.add(drain); stub.commands = known; - eventually(() -> stub.acknowledgements == 1); + int beforeAck = stub.acknowledgements; + stub.desiredState = "future-state"; stub.desiredRevision = 2; + assertThrows(ExecutionException.class, () -> client.readiness().get(10, TimeUnit.SECONDS)); + assertEquals(0, host.admissions, "NXS has no per-join provider state"); + assertTrue(stub.appliedRevision < 2, "Unknown state cannot be acknowledged"); + stub.desiredState = "draining"; + client.readiness().get(10, TimeUnit.SECONDS); + assertTrue(host.drains > 0); assertEquals(2, stub.appliedRevision); + assertTrue(stub.acknowledgements > beforeAck); client.stop().toCompletableFuture().get(10, TimeUnit.SECONDS); } } + @Test void outcomeOutageBacksOffWhileHeartbeatsContinue(@TempDir Path path) throws Exception { + try (IndependentProviderStub stub = new IndependentProviderStub()) { + FakeTransport host = new FakeTransport(); stub.failOutcomes = true; + var client = new ProviderClient(new ProviderClient.Configuration(URI.create(stub.origin), "nxs-admission-v1", "Example"), + new ProviderStateStore(path), host, () -> null, () -> new ProviderClient.Health(true, 10, 0, "nethernet", "fixture"), message -> {}); + try { + client.start().get(20, TimeUnit.SECONDS); + host.events.add(JsonParser.parseString("{\"ticketId\":\"fixture-ticket\",\"stage\":\"ticket.failed\",\"occurredAt\":\"2026-09-07T00:00:00Z\"}").getAsJsonObject()); + eventually(() -> stub.outcomeAttempts == 1); + int before = stub.heartbeats; + eventually(() -> stub.heartbeats >= before + 2); + assertEquals(1, stub.outcomeAttempts, "Outcome failure must back off independently of heartbeat"); + JsonObject saved = JsonParser.parseString(java.nio.file.Files.readString(path.resolve("provider-state.json"))).getAsJsonObject(); + assertEquals(1, saved.getAsJsonArray("pendingEvents").size()); + assertTrue(saved.get("profilePublishedAt").getAsLong() > 0); + } finally { stub.failOutcomes = false; client.stop().toCompletableFuture().get(10, TimeUnit.SECONDS); } + assertEquals(1, stub.events.size(), "Shutdown retries the durable outcome without losing it"); + } + } + @Test void localPersistenceFailureStopsPublicationAndClosesTransport(@TempDir Path path) throws Exception { try (IndependentProviderStub stub = new IndependentProviderStub()) { FakeTransport host = new FakeTransport(); diff --git a/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/ProviderJourneysTest.java b/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/ProviderJourneysTest.java index 370d5a2..dd625ca 100644 --- a/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/ProviderJourneysTest.java +++ b/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/ProviderJourneysTest.java @@ -39,7 +39,7 @@ private static ProviderClient client(IndependentProviderStub stub, Path director assertTrue(stub.keyAcknowledgements > 0); assertTrue(instance.readiness().get(10, TimeUnit.SECONDS).get("routable").getAsBoolean()); if (attach) assertEquals("london", result.getAsJsonObject("placement").getAsJsonObject("tags").get("location").getAsString()); - JsonObject event = new JsonObject(); event.addProperty("stage", "ticket.transport_established"); + JsonObject event = new JsonObject(); event.addProperty("stage", "ticket.data_channels_open"); event.addProperty("ticketId", "opaque-correlation"); event.addProperty("occurredAt", java.time.Instant.now().toString()); event.addProperty("reason", "connected"); event.addProperty("privatePayload", "must-not-be-persisted"); transport.events.add(event); long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); @@ -119,7 +119,7 @@ private static ProviderClient client(IndependentProviderStub stub, Path director try { JsonObject registration = resumed.start().get(20, TimeUnit.SECONDS); assertEquals(stub.registration.get("registrationId"), registration.get("registrationId")); - assertEquals(1, stub.registrations); assertEquals(1, stub.generation); + assertEquals(1, stub.registrations); assertEquals(2, stub.generation); assertTrue(resumed.readiness().get(10, TimeUnit.SECONDS).get("routable").getAsBoolean()); assertTrue(stub.keyAcknowledgements > 0, "Lost one-time key material is freshly provisioned"); } finally { resumed.stop().toCompletableFuture().get(10, TimeUnit.SECONDS); } diff --git a/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/admission/NativeAdmissionIntegrationTest.java b/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/admission/NativeAdmissionIntegrationTest.java index 7830e71..6cffb8b 100644 --- a/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/admission/NativeAdmissionIntegrationTest.java +++ b/external-signalling/src/test/java/org/cloudburstmc/netty/signalling/admission/NativeAdmissionIntegrationTest.java @@ -379,7 +379,7 @@ private static byte[] nominatedBinding(String username, String password) throws String incarnation = first.getAsJsonObject("statelessAdmission").get("incarnation").getAsString(); assertTrue(incarnation.matches("[0-9a-f]{32}")); var command = new com.google.gson.JsonObject();command.addProperty("kind","join-admission"); - assertEquals(org.cloudburstmc.netty.signalling.ProviderTransport.ApplyResult.REJECTED,transport.applyControl(command).toCompletableFuture().get()); + assertEquals(org.cloudburstmc.netty.signalling.ProviderTransport.ApplyResult.REJECTED,transport.applyState("join-admission").toCompletableFuture().get()); assertEquals(0,transport.channel().admissionStats().claims());assertEquals(0,transport.channel().nativeStats()[2]); assertEquals(creations,PeerConnection.nativeCreationAttempts()); transport.installTicketKeys(List.of(new org.cloudburstmc.netty.signalling.ProviderTransport.TicketKey("K001",TestSignallingProvider.SECRET,0,System.currentTimeMillis()+60_000),