Implement T2, T3, and part of T4 from API refactor plan by aidanfnv · Pull Request #413 · NVIDIA/flashdreams

aidanfnv · 2026-08-04T23:31:19Z

No description provided.

…api' into dev/aidanf/inference-runtime-api-T2-T3-T4 # Conflicts: # docs/inference_runtime_api_design.md

copy-pr-bot · 2026-08-04T23:31:22Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

fangjunzhou-nv · 2026-08-05T00:33:12Z

@@ -42,8 +42,8 @@ def close(self) -> None:
 class InferenceRuntime(Protocol):


This is not necessary related to the input system design in this PR, but I suggest we remove InferenceRuntime as it's only job is to spin up and tear down InferenceSession. We should just leave that to the Application.

fangjunzhou-nv · 2026-08-05T00:33:53Z

@@ -56,10 +56,10 @@ def close(self) -> None:
 class ModelAdapter(Protocol):


We should probably rename ModelAdapter to InferenceSessionSchema or something. It's just advertising the schemas for InferenceSession

jarcherNV · 2026-08-05T00:56:55Z

-    initial_inputs = mapping.map_initial_inputs(
-        user_inputs=user_inputs,
-        model_inputs=model_inputs,
+    initial_inputs = mapping.map_global_inputs(


I'm wondering if this is how we want it to behave.

The reference loop advances stateful input converters before the session starts. I think this canonicalizes the full session horizon for global inputs before runtime/session creation? But InputCanonicalizer converters are explicitly stateful and must be fed windows in session order.

KeyboardToDriverCommand.convert() mutates held-key state while reading events at canonical.py, line 177. If a real demo copied this loop, future key events could affect the initial state. Maybe the global conditioning should bypass canonicalization, or use a separate non-live startup path?

Not too sure about this one. Maybe @ArielG-NV or @fangjunzhou-nv may have an opinion. Or just throw this comment at codex and see what it thinks.

Bypassed canonicalization for global conditioning in cced4c7

jarcherNV · 2026-08-05T00:58:33Z

+registration rather than an application change. `InferenceInput` is what an
+`InferenceSession` actually receives.
+
+`InferenceInput` describes the data the model or inference pipeline actually


I think the implementation and the docs may be disagreeing with each other a little here, maybe just tell codex to do another once-over on the docs before you merge.

Addressed this after the fact in e1fa7d0

jarcherNV · 2026-08-05T01:01:15Z

+    )
+
+
+def check_mapping_compatibility(


The compatibility checking seems good, but not sure if it's wired into the primary adapter/mapping path yet?

This function looks useful, but I think ModelAdapter.default_input_mapping() returns only an InputMapping at interfaces.py line 80, and schema declaration is an optional refinement at.

Before demo migration, maybe the standard loop needs a clear way to get mapping schema(s) and run compatibility before loading checkpoints?

Not necessarily something that has to be fixed here, but we should keep a note of it when we get to demo migration maybe.

jarcherNV · 2026-08-05T01:04:39Z

+            device_kind="keyboard",
+            priority=priority,
+            consumes=(
+                UserInputCapability(


Looks like this consumes key_down / key_up. I think the older realtime helper uses keydown / keyup at input.py line 77 (flashdreams/flashdreams/serving/realtime/input.py:77), and one runtime test uses keyboard.keydown at test_inference_runtime_api.py, line 44.

I'm wondering if we need to standardise the names yet, or leave that for later. I imagine the demos will need a minimal event catalog or something.

ArielG-NV

Initial review; questions and suggestions

ArielG-NV · 2026-08-05T01:36:05Z

+```python
+sana_wm_model = InferenceInputSchema(
+    description="sana-wm",
+    global_fields=(


these fields should potentially have a semantic type as well?

example: a "negative prompt" may be treated differently than a positive prompt, the underlying model_arch needs support for the semantic type of each (since each are technically processed uniquely depending on the underlying model_arch)

Turns out that part of the doc is just wrong about what a semantic_type entails.
name="negative_prompt" and name="prompt" would provide the negative vs positive distinction.
I will be renaming semantic_type to input_modality, and have corrected the docs, see #421

name="negative_prompt" would distinguish a negative prompt from a positive name="prompt".
The docs describe semantic_type incorrectly, and included stuff like "path" which are not even supposed to be in that field.

I have changed "semantic type" to "input modality" and corrected the doc, see e1fa7d0 (accidentally committed that a bit prematurely).

ArielG-NV · 2026-08-05T01:37:56Z

+sana_wm_model = InferenceInputSchema(
+    description="sana-wm",
+    global_fields=(
+        InputField(name="prompt", lifecycle="cache_init"),


why are we using new terminology for the frequency of an input?

I suppose lifescycle is fine

but cache_init should be something like "global_conditioning"?

For "lifecycle", there was some partial attempt at separating the timeframe/frequency of input, frequency of consumption, and scope of usage (global vs step) and sometimes those share a label, sometimes they don't, which is needlessly confusing. "cache_init" describes a one-time global conditioning input.

global_fields is already the inputs for global conditioning (I will rename to global_conditioning_fields), so we do not need to specify "global_conditioning" again, and what makes "cache_init" different from other one-time global conditioning inputs is irrelevant here.

Honestly, "lifecycle" is not really fine here, it really confused myself and any agent I used here. Since the global vs step distinction is already handled elsewhere, I will rename it to frequency_consumed, where this example would have the value "once".

See #421 for the change

ArielG-NV · 2026-08-05T01:38:10Z

+        InputField(
+            name="camera_trajectory_c2w",
+            semantic_type="c2w_sequence",
+            lifecycle="rollout_binding",


what is a rollout_binding, why not something likeper_step or every_step?

Somehow in the process of implementing this, the global conditioning inputs got split into 5 "lifecycles", where rollout_binding represents "values bound after cache initialization but before AR steps, such as HY-WorldPlay action labels, camera tensors, and memory state".

That is not useful to specify, so rollout_binding is indeed a per-step global conditioning input. I will change this to per_step.

See #421 for the change

gtong-nv · 2026-08-05T16:21:23Z

+        InputField(name="scene_id", update_policy=SESSION_START_ONLY),
+    )
+)
+schema.unsupported_global_updates(


what is unsupported_global_updates ?

unsupported_global_updates() is a helper on InferenceInputSchema for mid-rollout global-conditioning update requests. It looks at the non-empty InferenceInput.global_conditioning payload passed to a step/update and returns the field names whose schema marks them SESSION_START_ONLY, meaning they can be supplied at session start but should not be swapped while the rollout is active.

It will be removed in #421

greptile-apps · 2026-08-05T17:01:21Z

Greptile Summary

The PR implements the runtime input refactor by separating raw user events, canonical device-independent modalities, and model-facing inference inputs.

Adds stateful device canonicalization, keyboard driving controls, and scripted replay modalities.
Adds declarative mapping schemas and source-to-model compatibility reporting.
Renames the model input API to global and per-step InferenceInput conditioning and updates protocols, tests, and design documentation.

Confidence Score: 4/5

The PR appears safe to merge, with a non-blocking device-priority issue where an idle keyboard can mask another active control source.

The new input layers are coherently implemented and tested, but the keyboard converter violates the canonicalizer’s documented idle-yield contract by always returning a neutral value.

Files Needing Attention: flashdreams/flashdreams/runtime/canonical.py

Important Files Changed

Filename	Overview
flashdreams/flashdreams/runtime/canonical.py	Adds device converter registration, priority selection, keyboard driving conversion, and scripted modalities; idle keyboard conversion can mask lower-priority devices.
flashdreams/flashdreams/runtime/inputs.py	Introduces canonical and inference input envelopes, capability schemas, phase-aware fields, and global-update metadata.
flashdreams/flashdreams/runtime/mapping.py	Defines canonical-to-inference mapping protocols and declarative compatibility reporting for single or composed mappings.
flashdreams/flashdreams/runtime/interfaces.py	Updates adapter, runtime, and session protocols to consume the new canonical and inference input contracts.
flashdreams/tests/test_runtime_canonical.py	Thoroughly covers conversion, state, priority, rebinding, and scripted replay, but omits an idle higher-priority keyboard competing with an active device.
flashdreams/tests/test_runtime_input_mapping.py	Covers input capabilities, schema matching, composed mappings, optional degradation, and global-update behavior.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    U[UserInputs<br/>raw events] --> C[InputCanonicalizer]
    C --> M[CanonicalInputs<br/>device-independent modalities]
    M --> I[InputMapping]
    G[Application-owned<br/>global conditioning] --> I
    I --> E[InferenceInput<br/>global + step]
    E --> S[InferenceSession]

_{Reviews (1): Last reviewed commit: "Align closer to diagrams" | Re-trigger Greptile}

greptile-apps · 2026-08-05T17:01:24Z

+        return DRIVER_COMMAND.value(
+            {
+                "throttle": 1.0 if held("throttle") else 0.0,
+                "brake": 1.0 if held("brake") else 0.0,
+                "steer": steer,
+                "stop": held("stop"),
+                "reverse": held("reverse"),
+            }
+        )


Idle keyboard masks active devices

If an idle keyboard converter has higher priority than another active driver_command converter, it still returns a neutral value instead of None, so InputCanonicalizer selects it and ignores the active device's steering, throttle, or braking input.

* WIP implementation of T2, T3, partial T4 * Fix issues found by Claude * Rewrite based on discussion, port after merge * doc update * doc updates * Update based on new diagrams * Align closer to diagrams

aidanfnv added 7 commits August 4, 2026 10:35

WIP implementation of T2, T3, partial T4

27e2d89

Fix issues found by Claude

d58ace2

Merge remote-tracking branch 'upstream/dev/jarcher/inference-runtime-…

7d5c901

…api' into dev/aidanf/inference-runtime-api-T2-T3-T4 # Conflicts: # docs/inference_runtime_api_design.md

Rewrite based on discussion, port after merge

4f39116

doc update

7a8c4d8

doc updates

e71b2f6

Update based on new diagrams

cb2056a

Align closer to diagrams

b6ed51a

fangjunzhou-nv reviewed Aug 5, 2026

View reviewed changes

jarcherNV reviewed Aug 5, 2026

View reviewed changes

ArielG-NV reviewed Aug 5, 2026

View reviewed changes

gtong-nv reviewed Aug 5, 2026

View reviewed changes

aidanfnv marked this pull request as ready for review August 5, 2026 16:58

aidanfnv merged commit ddae974 into dev/jarcher/inference-runtime-api Aug 5, 2026
3 checks passed

aidanfnv deleted the dev/aidanf/inference-runtime-api-T2-T3-T4 branch August 5, 2026 16:58

aidanfnv restored the dev/aidanf/inference-runtime-api-T2-T3-T4 branch August 5, 2026 16:58

greptile-apps Bot reviewed Aug 5, 2026

View reviewed changes

aidanfnv deleted the dev/aidanf/inference-runtime-api-T2-T3-T4 branch August 5, 2026 17:04

aidanfnv added a commit to aidanfnv/flashdreams that referenced this pull request Aug 5, 2026

Response to NVIDIA#413 (comment)

cf8c868

aidanfnv added a commit to aidanfnv/flashdreams that referenced this pull request Aug 5, 2026

Response to NVIDIA#413 (comment)

128c2b5

aidanfnv added a commit to aidanfnv/flashdreams that referenced this pull request Aug 6, 2026

Correct disagreements in docs from PR NVIDIA#413

4964503

		@@ -42,8 +42,8 @@ def close(self) -> None:
		class InferenceRuntime(Protocol):

		@@ -56,10 +56,10 @@ def close(self) -> None:
		class ModelAdapter(Protocol):

Uh oh!

Conversation

aidanfnv commented Aug 4, 2026

Uh oh!

copy-pr-bot Bot commented Aug 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArielG-NV left a comment

Choose a reason for hiding this comment

Uh oh!

ArielG-NV Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aidanfnv Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArielG-NV Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

greptile-apps Bot commented Aug 5, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Flowchart

Uh oh!

greptile-apps Bot Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ArielG-NV Aug 5, 2026 •

edited

Loading

aidanfnv Aug 6, 2026 •

edited

Loading

ArielG-NV Aug 5, 2026 •

edited

Loading