Skip to content

Clarify global conditioning input semantics #421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aidanfnv merged 1 commit into from
Aug 6, 2026
Merged

Clarify global conditioning input semantics #421

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 38 additions & 27 deletions docs/inference_runtime_api_design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ InferenceRuntime
|
v
InferenceSession
one rollout/stream: prompt/initial inputs, cache/state, current step, reset
one rollout/stream: global conditioning, cache/state, current step, reset
keeps per-run state from leaking across prompts, clients, or benchmark repeats
|
v
Expand Down Expand Up @@ -187,11 +187,11 @@ local model implementation, a Dynamo-like backend, or a hosted service.
| --- | --- | --- |
| Model/preset registry | Lists what can run: model/preset slugs, scenarios, capabilities, resource hints, and supported output modes. | Must remain cheap to query and must not load checkpoints. |
| App / integration / benchmark / transport | Owns the user-facing mode: CLI, native integration, WebRTC, benchmark, hosted request, or replay. | Supplies run setup, user inputs, model inputs, and output target selection. |
| User input library | Normalizes live or replayed controls into FlashDreams-supported user input events/windows. | Shared primitives for keyboard, reset, prompt/image updates, traces, and future scalar controls. |
| Input mapping | Converts user/app inputs plus initial model inputs into the model-specific inputs needed by the session. | A model adapter may provide a default mapping; runtimes, applications, benchmarks, and replay tools may override it without changing the model step. |
| User input library | Normalizes live or replayed controls into FlashDreams-supported user input events/windows. | Shared primitives for keyboard, reset, prompt/image selection, traces, and future scalar controls. |
| Input mapping | Converts user/app inputs plus global conditioning into the model-specific inputs needed by the session. | A model adapter may provide a default mapping; runtimes, applications, benchmarks, and replay tools may override it without changing the model step. |
| ModelRunner / standard loop | Orchestrates one run from setup through runtime initialization, stepping, output, metrics, and teardown. | Shared orchestration layer used by CLIs, benchmarks, MP4 runs, and simple realtime flows. |
| InferenceRuntime | Owns heavyweight lifecycle: distributed init, model construction, checkpoint loading, compile/capture, warmup, hosted-service connection, and teardown. | Long-lived reusable runtime created from `InferenceConfig`; lets FlashDreams load/warm once and create sessions sequentially unless the backend supports concurrency. |
| InferenceSession | Owns one rollout or stream: initial inputs, cache state, current step, reset behavior, step requirements, and step execution. | Per-rollout interface consumed by the standard loop; keeps state isolated across prompts, browser clients, replay scenarios, or benchmark repeats. |
| InferenceSession | Owns one rollout or stream: global conditioning, cache state, current step, reset behavior, step requirements, and step execution. | Per-rollout interface consumed by the standard loop; keeps state isolated across prompts, browser clients, replay scenarios, or benchmark repeats. |
| Model implementation / inference pipeline | Implements encode, model step, decode, cache updates, and model-specific optimizations. | FlashDreams wraps this boundary; it should not replace every model implementation. |
| Output target | Consumes generated outputs and handles presentation or persistence. | Separate from model execution so the same session can feed WebRTC, MP4, benchmark, or headless output. |
| Metrics, artifacts, and profiling | Records timings, memory, quality data, logs, reports, traces, and optional NVTX ranges. | Shared observation layer for local runs, benchmarks, CI smoke, and hosted runs. |
Expand Down Expand Up @@ -298,8 +298,7 @@ uses:

- keyboard keydown/keyup events;
- reset requests;
- prompt update requests;
- image update requests;
- prompt or image selection/update events;
- future scalar controls such as throttle, brake, steer, or camera axes once an
integration needs them.

Expand Down Expand Up @@ -335,11 +334,11 @@ Examples of global conditioning include prompt, negative prompt, conditioning
frame, input video, scene id, HD map asset, camera calibration, initial camera
pose, seed, or model-specific fields.

Global conditioning is normally supplied when a session starts, but a non-empty
global slot on a mid-rollout input is an update request rather than a reset;
resetting rollout state is a separate `InferenceSession.reset()` call. Whether a
given value can be swapped mid-rollout is declared per field by
`InputField.update_policy`.
Global conditioning establishes session-global model state when a session
starts or resets. During an active rollout, a non-empty global-conditioning
payload passed to `InferenceSession.step()` asks the session to update that
state when the model supports it. Reset remains a separate explicit session
method.

Examples of per-step conditioning include frame timestamps, pose segments,
camera trajectory chunks, rendered HD map frames, conditioning video windows,
Expand All @@ -349,16 +348,17 @@ Inference input payloads should use semantic names, not only modality names. For
example, a first frame and an HD map frame should be distinct inputs even if
both are image-like values.

Model input metadata may also include a lightweight lifecycle label, such as
runtime config, cache initialization, rollout binding, per-step input, or
session update. This should remain query metadata, not model-specific tensor
validation.
Model input names, payload kinds, semantic-type hints, and schema metadata
should be open-ended. Supported integrations such as SANA-WM, LingBot,
Omnidreams, and future external adapters may need different semantic fields.
Adding a new model should usually mean adding adapter-owned schema declarations
and mappings, not changing a central FlashDreams enum.

Model input names, payload kinds, lifecycle labels, and schema metadata should
be open-ended. Supported integrations such as SANA-WM, LingBot, Omnidreams, and
future external adapters may need different semantic fields. Adding a new model
should usually mean adding adapter-owned schema declarations and mappings, not
changing a central FlashDreams enum.
Consumption cadence is a separate hint from input scope. A field may be
provided through global conditioning because it is session-global state, while
the adapter consumes or slices it during every step. That can be recorded as
`frequency_consumed` metadata without changing whether the field belongs in
`global_conditioning_fields` or `step_fields`.

For interactive runs, most `InferenceInput` values will be global conditioning
plus per-step inputs produced by input mapping. For MP4 generation and benchmarking, the API
Expand All @@ -374,7 +374,7 @@ These schemas are not meant to be a rich type system or a replacement for
model-specific validation. They should be just enough to answer:

- what can this app, transport, trace, or benchmark source provide?
- what does this model require before startup and at each step?
- what does this model require before session start and at each step?
- can this event source drive this model with the selected mapping?

The purpose is to fail early before expensive model initialization, produce
Expand All @@ -386,7 +386,7 @@ coordinate frame, units, rough shape summary, accepted file suffixes, schema
URI, model family, or source/transport details. Metadata should help humans and
adapter selection code, but compatibility should still be based on the declared
event capabilities, semantic model fields, payload representation hints, and
lifecycle labels.
schema phases. Consumption-cadence hints are descriptive and adapter-owned.

For simple CLI text-to-video or image-to-video runs, `UserInputSchema` can be
trivial or omitted because there may be no live controls. `InferenceInputSchema` is
Expand Down Expand Up @@ -478,6 +478,14 @@ declare user inputs, declare model inputs, and provide a default mapping, while
the runtime owns transport, event validation, timestamping, input queue/window
selection, output delivery, and optional overrides.

`StepRequest` and `StepResult` are per-step runtime messages, not declarative
schemas. `InferenceSession.next_step_request()` returns a `StepRequest` to say
which step is next, which user-input time window to map, and whether this step
has any narrower `InferenceInputSchema` than the session default. The runner or
application then builds an `InferenceInput` and calls `InferenceSession.step()`,
which returns a `StepResult` carrying the generated output, output timing,
metrics, and step metadata.

Examples:

- T2V mapping validates a prompt and creates no per-step control inputs.
Expand Down Expand Up @@ -505,7 +513,7 @@ A run should:
profiling, and optional scenario setup.
3. Validate that the event source and mapping can drive the selected model.
4. Initialize the runtime.
5. Start a session from initial model inputs.
5. Start a session from global conditioning inputs.
6. For each step, ask the session what it needs, gather live or fixed inputs,
build step model inputs, run the session step, route outputs, and record
metrics.
Expand Down Expand Up @@ -553,8 +561,9 @@ generation, benchmarks, regression testing, and autotune.

Two replay levels should be supported:

- user-event replay: records timestamped key events, prompt updates, image
updates, reset events, and timing, then runs normal input mapping;
- user-event replay: records timestamped key events, prompt or image
selection/update events, reset events, and timing, then runs normal input
mapping;
- model-input replay: records or defines already-mapped per-step model inputs
for stricter model-level regression tests.

Expand Down Expand Up @@ -667,8 +676,10 @@ registry, standard loop, concrete output modes, or model migrations:
- The model-specific integration boundary is named `ModelAdapter`.
- Heavyweight lifecycle is split into `InferenceRuntime` and
`InferenceSession`.
- Step data carriers are named `StepRequest` and `StepResult`; a session returns
`None` from `next_step_request()` when the rollout is complete.
- Step data carriers are named `StepRequest` and `StepResult`. They are runtime
messages around one call to `InferenceSession.step()`, not schema
declarations; a session returns `None` from `next_step_request()` when the
rollout is complete.
- Raw inputs use `UserInputs`, canonicalized inputs use `CanonicalInputs`, and
model-facing inputs use `InferenceInput`.
Both remain lightweight payload envelopes with shallow read-only mappings.
Expand Down
97 changes: 48 additions & 49 deletions docs/inference_runtime_inputs_implementation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,70 +46,60 @@ that touches no application, mapping, or model code.

This path covers **live user control only**. Global conditioning is
application-owned data and reaches `InferenceInput` directly, without passing
through canonicalization or a device converter. An application that wants a
trigger key to swap the prompt reads that as ordinary canonical control input
and updates its own global conditioning in response.
through canonicalization or a device converter. Session start/reset establishes
that global conditioning. During an active rollout, a non-empty
`global_conditioning` payload passed to `step()` requests an update of the
session-global state when the model supports it.

## Conditioning Slots

Both the canonical and encoded layers split into two slots, and the split means
the same thing at each:
The encoded layer splits model-facing inputs into two slots:

- **global conditioning** — conditions the whole rollout: prompt, conditioning
frame, scene. Normally supplied at session start.
- **global conditioning** — session-global model state: prompt, conditioning
frame, scene.
- **per-step conditioning** — needed to generate the next chunk or frame:
steering, HD map frames, camera trajectory.

`InputPhase` is `Literal["global", "step"]`. The axis names *which slot*, not
*when the value may arrive* — see the next section.
`InputPhase` is `Literal["global_conditioning", "step"]`. The phase names the
`InferenceInput` slot the caller provides.

## Global Conditioning Updates Are Not Resets
`InputField.frequency_consumed` is independent query metadata. It says how the
adapter consumes a field internally, such as `once` or `per_step`; it does not
decide whether the caller provides the field through `global_conditioning` or
`step`.

A non-empty global slot on a mid-rollout `InferenceInput` is an **update
request**. The session should apply it when the model supports doing so.
Resetting rollout state is a separate, explicit `InferenceSession.reset()` call.
The motivating case is changing prompt and conditioning frame mid-run to change
the weather in an Omnidreams rollout.
## Global Conditioning Is Session-Global State

```python
from flashdreams.runtime import InferenceInput

steady_state = InferenceInput(step={"steering": 0.25})
assert not steady_state.requests_global_update

changed_weather = steady_state.with_global_update({"prompt": "heavy rain"})
assert changed_weather.requests_global_update
```

Because `with_step()` carries the global slot through unchanged, use
`without_global_update()` for the steady-state case; otherwise every step looks
like an update request.

Whether a value can actually be swapped mid-rollout is declared per field:
`InferenceInput.global_conditioning` carries session-scoped inputs. A runtime
passes those values to `InferenceRuntime.start_session()` or to
`InferenceSession.reset()` when the backend supports resetting a rollout.
During an active rollout, passing a non-empty `global_conditioning` payload to
`InferenceSession.step()` asks the session to update that session-global state.
The model/session owns whether that update is supported.

```python
from flashdreams.runtime import SESSION_START_ONLY, InferenceInputSchema, InputField
from flashdreams.runtime import InferenceInput, InferenceInputSchema, InputField

schema = InferenceInputSchema(
global_fields=(
InputField(name="prompt", update_policy="step_boundary"),
InputField(name="scene_id", update_policy=SESSION_START_ONLY),
global_conditioning_fields=(
InputField(name="prompt"),
InputField(name="scene_id"),
)
)
schema.unsupported_global_updates(
InferenceInput(global_conditioning={"prompt": "heavy rain", "scene_id": "town_02"})
schema.require_global_conditioning(
InferenceInput(global_conditioning={"prompt": "drive", "scene_id": "town_02"})
)
# ("scene_id",)
```

`SESSION_START_ONLY` is the one reserved `update_policy` token. Everything else
in that vocabulary, and all of `lifecycle`, is open and adapter-owned; this layer
only carries it as queryable metadata.
step_with_prompt_update = InferenceInput(
global_conditioning={"prompt": "heavy rain"},
step={"steering": 0.0},
)
```

Steady-state steps must leave the global slot empty; otherwise every step reads
as an update request. Converters emit every window, because live control is
level-triggered: a key held across a step emits no events but still means full
throttle.
Per-step conditioning is different: those values are supplied through
`InferenceInput.step` for each generated chunk or frame. Converters still emit
every window, because live control is level-triggered: a key held across a step
emits no events but still means full throttle.

## Raw Inputs

Expand Down Expand Up @@ -192,8 +182,9 @@ device does not resume from stale state.
## Mapping And Compatibility

`InputMapping` is the canonical-to-encoded boundary. `InputMappingSchema` is its
declarative surface: `consumes` names canonical modalities; `produces_global`
and `produces_step` name the `InferenceInput` fields it can build.
declarative surface: `consumes` names canonical modalities;
`produces_global_conditioning` and `produces_step` name the `InferenceInput`
fields it can build.

`InputMapping.validate()` raises, which fails a run late and cannot say *which*
optional model input a source would enable or *which* missing modality makes a
Expand Down Expand Up @@ -230,15 +221,23 @@ registered later, with no change to the mapping or the model schema.
`undeclared_inference_inputs()` reports payload keys a mapping produced but did
not declare, which keeps hand-written schemas honest as the code drifts.

`StepRequest` and `StepResult` sit around a single `InferenceSession.step()`
call. They are not schema declarations. A session returns `StepRequest` from
`next_step_request()` to name the next step, optionally provide a narrower
`InferenceInputSchema`, and request a `TimeWindow` of user inputs. The runner
then builds `InferenceInput` and calls `step()`, which returns a `StepResult`
for the output target and metrics recorder.

## What This Does Not Validate

The schemas intentionally avoid becoming a rich type system. These remain the
responsibility of the model adapter, runtime, session, or mapping:

- tensor shape and dtype, image decode details;
- camera coordinate systems, pose and timestamp units;
- prompt-embedding swap mechanics;
- whether a model can actually apply a declared update policy at runtime;
- prompt-embedding mechanics;
- whether a model can actually apply a requested global-conditioning update;
- enforcing consumption-cadence metadata;
- deep validation of scene, HD map, or actor-state data.

The layer answers "can this source plausibly drive this model through this
Expand Down
Loading
Loading