Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/skills/contribute-api/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ runtime or bindings.
- Keep every affected public surface in parity
- `RunRequest.relay_session_root` is forwarded as the typed `AgentRunRequest.relay_session_root` field. Keep Rust, Python, TypeScript, and schema representations in parity. Context keys do not control Relay propagation; preserve the adapter fallback for unusable UUIDs.
- OpenShell task commands serialize public SDK config/request/handle/result mappings. Keep forwarding buffered and preserve task-generated IDs, normalized failed results, telemetry references, and declared artifacts. Do not normalize an SDK result as an adapter result. Task paths remain POSIX strings; host artifact destinations stay separate. Unknown outcomes bind the immutable sandbox identity through reattachment and must not be replayed.
- Runtime discovery uses the typed public `Fabric.runtime_status()` operation and `sdk/runtime-status-report` schema. Keep `fabric-agent-ctl status` delegated to that SDK operation and OpenShell status routed through the existing provider command transport. Preserve recorded lifecycle state separately from health observations, use null handles for stopped records, and exclude environment variables and arbitrary provider metadata from discovery snapshots. Each client caps all runtime records, including pending starts, at 128; evict only stopped records and reject full-capacity starts with `runtime_limit_reached` before runtime effects.
- Update docs and examples in the same branch

## Minimum Acceptance
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ NeMo Fabric separates consumers from execution targets through a stable, version
- [Installation](docs/getting-started/install.mdx): Install the runtime, adapters, harness dependencies, and telemetry components, and choose a deployment scenario.
- [Adapter catalog](sdk/python/nemo-fabric-adapter-catalog/README.md): Inspect bundled adapter metadata on a host without installing harness SDKs.
- [Host admission](docs/sdk/python.mdx#inspect-metadata-on-a-separate-host): Validate requested capabilities and reject host/task descriptor drift before execution.
- [NeMo Fabric runtime server](sdk/python/nemo-fabric-runtime/src/nemo_fabric/runtime_server/README.md): Manage runtimes and check server responsiveness inside one sandbox through `fabric-agent-ctl` and one private local socket. Unconfirmed cleanup retains sandbox ownership and blocks replacement servers.
- [NeMo Fabric runtime server](sdk/python/nemo-fabric-runtime/src/nemo_fabric/runtime_server/README.md): Manage runtimes, rediscover their recorded status and safe handles, and check server responsiveness inside one sandbox through `fabric-agent-ctl` and one private local socket. Each SDK client keeps at most 128 runtime records, evicting stopped records first and rejecting new starts when active, failed, and pending runtimes fill the limit. Unconfirmed cleanup retains sandbox ownership and blocks replacement servers.
- [Health checks](docs/sdk/health-checks.mdx): Use `Fabric.check()` for OpenShell runtime-server checks, agent runtime checks, and prepared or attached OpenClaw Gateway service checks. Refer to the [adapter support table](docs/sdk/health-checks.mdx#native-evidence-by-adapter) for supported levels and native evidence. OpenShell server and runtime requests map to `fabric-agent-ctl check`; service checks use the local SDK.
- [Documentation overview](docs/about-nemo-fabric/overview.mdx): Understand the execution model and choose an interface.
- [Adapter reference](adapters/README.md): Compare adapter configuration, runtime, and observability support.
Expand Down
93 changes: 89 additions & 4 deletions crates/fabric-core/src/environment.rs
Original file line number Diff line number Diff line change
Expand Up @@ -497,7 +497,7 @@ where
}
}

// A dedicated provider process handles each health check and exits after the response.
// A dedicated provider process handles each read-only command and exits after the response.
fn request_check_provider_process<T>(
command: OsString,
operation: ProviderOperation<'_>,
Expand All @@ -506,6 +506,13 @@ fn request_check_provider_process<T>(
where
T: for<'de> Deserialize<'de>,
{
let is_check = matches!(
&operation,
ProviderOperation::RuntimeCommand {
command: TaskCommand::Check { .. },
..
}
);
let started = Instant::now();
let budget = Duration::try_from_secs_f64(timeout_seconds).unwrap_or(Duration::MAX);
let remaining = || {
Expand All @@ -531,7 +538,9 @@ where
})
.map_err(FabricError::SerializeJson)?;
let seconds = remaining()?.as_secs_f64();
request["command"]["timeout_seconds"] = Value::from(seconds);
if is_check {
request["command"]["timeout_seconds"] = Value::from(seconds);
}
request["timeout_seconds"] = Value::from(seconds.ceil().clamp(1.0, 320.0) as u64);
let mut encoded = serde_json::to_vec(&request).map_err(FabricError::SerializeJson)?;
if encoded.len() > MAX_PROVIDER_REQUEST_BYTES {
Expand Down Expand Up @@ -567,13 +576,30 @@ where
}
};
remaining()?;
if output.is_empty() || output.len() > 16 * 1024 || output.last() != Some(&b'\n') {
if output.is_empty()
|| output.len()
> if is_check {
16 * 1024
} else {
MAX_PROVIDER_RESPONSE_BYTES
}
|| output.last() != Some(&b'\n')
{
return provider_error(
"runtime_command",
"provider_protocol_error",
"invalid health response framing".into(),
);
}
if !is_check {
crate::strict_json::validate(&output).map_err(|_| {
provider_failure(
"runtime_command",
"provider_protocol_error",
"invalid status response".into(),
)
})?;
}
let response: ProviderResponse = serde_json::from_slice(&output).map_err(|_| {
provider_failure(
"runtime_command",
Expand Down Expand Up @@ -635,11 +661,16 @@ where
let budget = *timeout_seconds;
return request_check_provider_process(command, operation, budget);
}
if matches!(task_command, TaskCommand::Status { .. }) {
return request_check_provider_process(command, operation, 10.0);
}
let operation_name = operation.name();
let request_id = new_id("environment-provider-request");
let environment_key = environment_resource_key(environment)?;
let (identity, is_start, is_stop) = match task_command {
TaskCommand::Check { .. } => unreachable!("health uses a separate provider process"),
TaskCommand::Check { .. } | TaskCommand::Status { .. } => {
unreachable!("reads use a separate provider process")
}
TaskCommand::Start { .. } => (request_id.clone(), true, false),
TaskCommand::Invoke { runtime_id, .. } => (runtime_id.clone(), false, false),
TaskCommand::Stop { runtime_id, .. } => (runtime_id.clone(), false, true),
Expand Down Expand Up @@ -1449,6 +1480,60 @@ mod tests {
fs::remove_dir_all(directory).expect("remove fixture");
}

#[test]
fn status_provider_does_not_wait_for_an_invocation_worker() {
let (directory, command) = provider_script(
r#"
import json, sys
for line in sys.stdin:
request = json.loads(line)
assert request['command']['operation'] == 'status'
assert set(request['command']) == {'operation', 'agent', 'runtime_id'}
print(json.dumps({'protocol_version': request['protocol_version'],
'request_id': request['request_id'], 'status': 'succeeded',
'output': {'seen': 'status'}}), flush=True)
"#,
);
let environment: EnvironmentHandle = serde_json::from_value(serde_json::json!({
"environment_id": "status-worker", "provider": "openshell",
"control_location": "in_env_control", "ownership": "caller_owned",
"connection": {"gateway": "https://status.invalid", "sandbox_id": new_id("sandbox")}
}))
.expect("environment fixture");
let key = serde_json::json!([environment_resource_key(&environment).unwrap(), "runtime-1"])
.to_string();
let process = runtime_provider_process(&key, &command, "runtime_command").expect("worker");
let guard = process.lock().unwrap();
let (sender, receiver) = mpsc::channel();
thread::scope(|scope| {
scope.spawn(|| {
let task = TaskCommand::Status {
agent: "demo".to_string(),
runtime_id: "runtime-1".to_string(),
};
let result = request_runtime_provider_process::<Value>(
command,
ProviderOperation::RuntimeCommand {
environment: &environment,
command: &task,
timeout_seconds: 10,
},
);
sender.send(result).unwrap();
});
let result = receiver.recv_timeout(Duration::from_secs(2));
drop(guard);
assert_eq!(
result
.expect("status must bypass the invocation mutex")
.expect("status output")["seen"],
"status"
);
});
retire_runtime_provider_process(&key, &process);
fs::remove_dir_all(directory).expect("remove fixture");
}

#[test]
fn health_provider_cleanup_is_bounded_with_inherited_pipes() {
for pipe in ["stdout", "stderr", "both"] {
Expand Down
1 change: 1 addition & 0 deletions crates/fabric-core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ pub mod provenance;
pub mod resource_snapshot;
pub mod runtime;
pub mod schema;
mod strict_json;
pub mod system_stats;
mod task_command;
mod time;
Expand Down
39 changes: 39 additions & 0 deletions crates/fabric-core/src/runtime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,45 @@ pub(crate) fn remote_check_agent(
openshell::check_agent(environment, runtime_id)
}

// Internal transport entrypoint for the typed Python SDK status operation.
#[doc(hidden)]
pub fn runtime_status_environment(
environment: &EnvironmentHandle,
agent: &str,
runtime_id: &str,
) -> Result<Value> {
if environment.provider != "openshell"
|| environment.control_location != ControlLocation::InEnvControl
|| [agent, runtime_id]
.iter()
.any(|value| value.trim().is_empty() || value.contains('\0'))
{
return Err(FabricError::InvalidConfig {
field: "runtime_status".to_string(),
reason: "a valid runtime target and OpenShell environment are required".to_string(),
});
}
let command = crate::task_command::TaskCommand::Status {
agent: agent.to_string(),
runtime_id: runtime_id.to_string(),
};
match crate::environment::command_runtime(environment, &command, 10)? {
crate::task_command::TaskCommandResponse::Returned { value } => Ok(value),
crate::task_command::TaskCommandResponse::Rejected { error } => {
Err(FabricError::EnvironmentProviderOperation {
provider: "openshell".to_string(),
operation: "status".to_string(),
code: if error.effects == "none" {
error.code
} else {
"invalid_response".to_string()
},
message: "The runtime status request was rejected.".to_string(),
})
}
}
}

static NEXT_ID: AtomicU64 = AtomicU64::new(1);
const ADAPTER_PYTHON_ENV: &str = "ADAPTER_PYTHON";
const VIRTUAL_ENV_ENV: &str = "VIRTUAL_ENV";
Expand Down
48 changes: 47 additions & 1 deletion crates/fabric-core/src/schema.rs
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ pub enum SchemaName {
EnvironmentReference,
/// Runtime handle schema.
RuntimeHandle,
/// Recorded SDK runtime state and sanitized discovery handle.
RuntimeStatusReport,
/// Caller-supplied reference to an already-running service.
ServiceReference,
/// Prepared or attached service handle schema.
Expand All @@ -84,7 +86,7 @@ pub enum SchemaName {

impl SchemaName {
/// All public schemas in stable output order.
pub const ALL: [Self; 28] = [
pub const ALL: [Self; 29] = [
Self::ExecutionEvidence,
Self::CheckRequest,
Self::CheckReport,
Expand All @@ -104,6 +106,7 @@ impl SchemaName {
Self::EnvironmentHandle,
Self::EnvironmentReference,
Self::RuntimeHandle,
Self::RuntimeStatusReport,
Self::ServiceReference,
Self::ServiceHandle,
Self::InvocationHandle,
Expand Down Expand Up @@ -137,6 +140,7 @@ impl SchemaName {
Self::EnvironmentHandle => "environment-handle",
Self::EnvironmentReference => "environment-reference",
Self::RuntimeHandle => "runtime-handle",
Self::RuntimeStatusReport => "runtime-status-report",
Self::ServiceReference => "service-reference",
Self::ServiceHandle => "service-handle",
Self::InvocationHandle => "invocation-handle",
Expand Down Expand Up @@ -200,6 +204,7 @@ impl SchemaName {
"environment-handle" | "environment_handle" => Ok(Self::EnvironmentHandle),
"environment-reference" | "environment_reference" => Ok(Self::EnvironmentReference),
"runtime-handle" | "runtime_handle" => Ok(Self::RuntimeHandle),
"runtime-status-report" | "runtime_status_report" => Ok(Self::RuntimeStatusReport),
"service-reference" | "service_reference" => Ok(Self::ServiceReference),
"service-handle" | "service_handle" => Ok(Self::ServiceHandle),
"invocation-handle" | "invocation_handle" => Ok(Self::InvocationHandle),
Expand Down Expand Up @@ -246,6 +251,7 @@ pub fn generate_schema(schema: SchemaName) -> Result<Value> {
SchemaName::EnvironmentHandle => to_value(schema_for!(EnvironmentHandle)),
SchemaName::EnvironmentReference => to_value(schema_for!(EnvironmentReference)),
SchemaName::RuntimeHandle => to_value(schema_for!(RuntimeHandle)),
SchemaName::RuntimeStatusReport => runtime_status_report_schema(),
SchemaName::ServiceReference => to_value(schema_for!(ServiceReference)),
SchemaName::ServiceHandle => to_value(schema_for!(ServiceHandle)),
SchemaName::InvocationHandle => to_value(schema_for!(InvocationHandle)),
Expand All @@ -258,6 +264,45 @@ pub fn generate_schema(schema: SchemaName) -> Result<Value> {
}
}

/// Generate the Python-owned status wire contract from the shared handle schema.
/// Status lookup remains an SDK operation; Rust only transports its JSON result.
fn runtime_status_report_schema() -> Result<Value> {
let mut handle = generate_schema(SchemaName::RuntimeHandle)?;
let mut definitions = handle.as_object_mut().unwrap().remove("$defs").unwrap();
handle.as_object_mut().unwrap().remove("$schema");
handle["additionalProperties"] = Value::Bool(false);
let environment = definitions["EnvironmentHandle"]["properties"]
.as_object_mut()
.unwrap();
for field in ["env", "connection", "metadata"] {
environment.remove(field);
}
definitions["RuntimeHandle"] = handle;
let identity = serde_json::json!({
"type": "string", "minLength": 1, "pattern": "\\S",
"not": {"pattern": "\\u0000"}
});
Ok(serde_json::json!({
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "RuntimeStatusReport",
"description": "Recorded lifecycle state and a sanitized discovery handle, not a health observation.",
"type": "object",
"additionalProperties": false,
"required": ["runtime_id", "agent_name", "status", "handle", "server_state"],
"properties": {
"runtime_id": identity,
"agent_name": identity,
"status": {"enum": ["active", "stopped", "failed"]},
"handle": {"anyOf": [{"$ref": "#/$defs/RuntimeHandle"}, {"type": "null"}]},
"server_state": {"enum": ["running", "stopping", "stopped", "unknown", null]}
},
"if": {"properties": {"status": {"const": "stopped"}}},
"then": {"properties": {"handle": {"type": "null"}}},
"else": {"properties": {"handle": {"$ref": "#/$defs/RuntimeHandle"}}},
"$defs": definitions
}))
}

/// Generate all schemas keyed by stable schema name.
pub fn generate_all_schemas() -> Result<BTreeMap<String, Value>> {
SchemaName::ALL
Expand Down Expand Up @@ -369,6 +414,7 @@ mod tests {
SchemaName::EnvironmentHandle,
SchemaName::EnvironmentReference,
SchemaName::RuntimeHandle,
SchemaName::RuntimeStatusReport,
SchemaName::InvocationHandle,
SchemaName::RunRequest,
SchemaName::RunResult,
Expand Down
65 changes: 65 additions & 0 deletions crates/fabric-core/src/strict_json.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

//! Validate JSON object keys before conversion to lossy map representations.

use std::collections::BTreeSet;
use std::fmt;

use serde::de::{self, MapAccess, SeqAccess, Visitor};
use serde::{Deserialize, Deserializer};

pub(crate) fn validate(encoded: &[u8]) -> serde_json::Result<()> {
serde_json::from_slice::<UniqueKeys>(encoded).map(|_| ())
}

struct UniqueKeys;

impl<'de> Deserialize<'de> for UniqueKeys {
fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
deserializer.deserialize_any(Self)
}
}

impl<'de> Visitor<'de> for UniqueKeys {
type Value = Self;

fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str("JSON without duplicate object fields")
}

fn visit_bool<E: de::Error>(self, _: bool) -> Result<Self, E> {
Ok(self)
}
fn visit_i64<E: de::Error>(self, _: i64) -> Result<Self, E> {
Ok(self)
}
fn visit_u64<E: de::Error>(self, _: u64) -> Result<Self, E> {
Ok(self)
}
fn visit_f64<E: de::Error>(self, _: f64) -> Result<Self, E> {
Ok(self)
}
fn visit_str<E: de::Error>(self, _: &str) -> Result<Self, E> {
Ok(self)
}
fn visit_unit<E: de::Error>(self) -> Result<Self, E> {
Ok(self)
}

fn visit_seq<A: SeqAccess<'de>>(self, mut sequence: A) -> Result<Self, A::Error> {
while sequence.next_element::<Self>()?.is_some() {}
Ok(self)
}

fn visit_map<A: MapAccess<'de>>(self, mut mapping: A) -> Result<Self, A::Error> {
let mut keys = BTreeSet::new();
while let Some(key) = mapping.next_key::<String>()? {
if !keys.insert(key) {
return Err(de::Error::custom("duplicate JSON field"));
}
mapping.next_value::<Self>()?;
}
Ok(self)
}
}
4 changes: 4 additions & 0 deletions crates/fabric-core/src/task_command.rs
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ pub(crate) enum TaskCommand {
runtime_id: String,
request: Value,
},
Status {
agent: String,
runtime_id: String,
},
Stop {
agent: String,
runtime_id: String,
Expand Down
Loading
Loading