diff --git a/AGENTS.md b/AGENTS.md index 19e4071f1..cbc42790f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -67,7 +67,7 @@ Seven terms have exactly one meaning each, everywhere in this repository: code c - No Engine test sleeps, reads the wall clock, waits with a timeout, or draws entropy from the OS. - Pseudo-random test input comes only from a fixed seed that the failure prints. - Temporary directory names come from the test's name, the process id, and a counter. -- Source directories are flat by default. A subdirectory of source files must contain at least three files, and its count includes the files in its nested subdirectories; one or two files belong beside the parent module as `foo-bar.rs` (parent stem, dash, kebab label), wired with an explicit path attribute so the module name stays clean: `#[path = "foo-bar.rs"] mod bar;`. The two forms are convertible in both directions: when a `foo-*.rs` sibling group grows to three files, rehydrate it into a `foo/` subdirectory in standard module layout (`foo/bar.rs` beside `foo.rs`) and drop the path attributes; when a subdirectory shrinks below three files, flatten it back to kebab siblings. Both conversions apply at each level of nesting. Apply whichever conversion applies when you touch files in a group on the wrong side of the line. Top-level `tests/` and `benches/` trees are exempt; they follow Cargo target conventions. +- A module with more than one file keeps its files in a directory named after it, in the standard module layout (`foo.rs` beside `foo/bar.rs`). A directory may hold any number of files, one included, and a module's unit tests are one of them (`foo/tests.rs`). Never move files between a flat sibling form and a directory because of how many files a group holds, and never change a group's layout only to bring it in line with this rule. Existing `foo-bar.rs` siblings wired with `#[path = "foo-bar.rs"] mod bar;` stay as they are, and new code adds no path attribute for layout. Top-level `tests/` and `benches/` trees follow Cargo target conventions. ## SPA and CSS Rules diff --git a/Cargo.lock b/Cargo.lock index 23ca7f255..10685000d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4062,6 +4062,7 @@ dependencies = [ name = "plugin-std" version = "0.3.0" dependencies = [ + "regex", "runvil-plugin", "serde", "serde_json", diff --git a/README.md b/README.md index c1b8bf38b..349f69517 100644 --- a/README.md +++ b/README.md @@ -26,16 +26,10 @@ This is [`prompts/greet.md`](prompts/greet.md), one of the sample prompts in thi name: greet description: Greet the named input using a Lua-computed value runvil: 0 -models: - writer: {} --- # Greet -```lua -models.default("writer") -``` - Computes a greeting from the input in Lua, substitutes it into the prose, and has the model echo it. @@ -52,7 +46,7 @@ return models.infer(prose) ``` ```` -The frontmatter declares one model role, `writer`, and the Lua under the title makes it the default. Given the input `Ada`, the first block in `## Main` builds `Hello, Ada!`, the placeholder puts it into the prose, and `models.infer(prose)` sends that prose to the model. The model's reply is the run's result. The paragraph under the title never reaches the model, because no Lua reads it, so it works as a comment. +The prompt declares no models, so `models.infer(prose)` runs on the Host's default model. Given the input `Ada`, the first block in `## Main` builds `Hello, Ada!`, the placeholder puts it into the prose, and `models.infer(prose)` sends that prose to the model. The model's reply is the run's result. A prompt can pick another of the five roles (`default`, `strong`, `small`, `fast`, `creative`) with `models.use(role)`, and the Host decides which model fills each role. The paragraph under the title never reaches the model, because no Lua reads it, so it works as a comment. ## How it fits together diff --git a/crates/harness-gateway-client/src/failure.rs b/crates/harness-gateway-client/src/failure.rs index d311a07da..4cb06537e 100644 --- a/crates/harness-gateway-client/src/failure.rs +++ b/crates/harness-gateway-client/src/failure.rs @@ -101,14 +101,13 @@ mod tests { } #[test] - fn a_decoded_parts_refusal_is_a_retryable_malformed_reply_with_its_specific() { + fn a_decoded_parts_refusal_is_a_malformed_reply_with_its_specific() { use runvil::model::ToolCall; let refusal = ToolCall::from_parts(" ", "fetch", serde_json::json!({})) .expect_err("a blank id is refused"); assert_eq!(refusal.kind(), CompletionErrorKind::InvalidParts); let reply = decoded(refusal); assert_eq!(reply.kind(), CompletionErrorKind::MalformedResponse); - assert!(reply.is_retryable()); assert_eq!( reply.to_string(), "the model backend sent a reply that could not be understood: tool call id was blank" diff --git a/crates/harness-gateway-client/src/transport/tests/limits.rs b/crates/harness-gateway-client/src/transport/tests/limits.rs index 52ea06116..83e11c169 100644 --- a/crates/harness-gateway-client/src/transport/tests/limits.rs +++ b/crates/harness-gateway-client/src/transport/tests/limits.rs @@ -22,7 +22,6 @@ async fn complete_on_a_disabled_client_is_an_unavailable_error() { err.to_string(), "model access is turned off or not configured" ); - assert!(!err.is_retryable()); } #[tokio::test] @@ -138,7 +137,6 @@ async fn a_request_past_the_timeout_is_a_timeout_failure() { CompletionErrorKind::Timeout, "the timeout must be recognizable: {err:?}" ); - assert!(err.is_retryable()); } /// Reads one request through its body, so answering and closing never @@ -270,7 +268,6 @@ async fn a_stream_that_stalls_after_the_headers_is_a_timeout_failure() { CompletionErrorKind::Timeout, "the timeout must be recognizable: {err:?}" ); - assert!(err.is_retryable()); } #[tokio::test] @@ -337,7 +334,6 @@ async fn a_refused_connection_is_a_transport_failure() { err.to_string(), "the connection to the model backend failed" ); - assert!(err.is_retryable()); assert!(std::error::Error::source(&err).is_some()); } diff --git a/crates/harness-gateway-client/src/wire/classify-tests.rs b/crates/harness-gateway-client/src/wire/classify-tests.rs index 04d9bf44f..90c941f8b 100644 --- a/crates/harness-gateway-client/src/wire/classify-tests.rs +++ b/crates/harness-gateway-client/src/wire/classify-tests.rs @@ -340,9 +340,15 @@ fn each_http_kind_carries_its_fixed_phrase_and_the_status_suffix() { } #[test] -fn a_429_is_retryable_now() { - assert!(classify_http_failure(429, "slow down").is_retryable()); - assert!(!classify_http_failure(429, "insufficient_quota").is_retryable()); +fn a_429_is_rate_limited_unless_its_body_names_a_spent_quota() { + assert_eq!( + classify_http_failure(429, "slow down").kind(), + CompletionErrorKind::RateLimited + ); + assert_eq!( + classify_http_failure(429, "insufficient_quota").kind(), + CompletionErrorKind::QuotaExhausted + ); } #[test] @@ -406,5 +412,4 @@ fn an_in_stream_envelope_that_matches_no_rule_stays_a_transport_failure() { "the connection to the model backend failed" ); assert_eq!(error.detail(), Some("upstream\\ndied")); - assert!(error.is_retryable()); } diff --git a/crates/harness-gateway-client/src/wire/request-tests.rs b/crates/harness-gateway-client/src/wire/request-tests.rs index df3283b69..d5e09d91a 100644 --- a/crates/harness-gateway-client/src/wire/request-tests.rs +++ b/crates/harness-gateway-client/src/wire/request-tests.rs @@ -5,15 +5,18 @@ //! constructor, so [`tool_round`] takes both from a facade `Run` that went //! through one tool round. +use std::collections::BTreeMap; use std::fmt::Write as _; use std::num::NonZeroU32; use std::sync::Arc; use runvil::effect::{Effect, EffectAnswer, IssuedEffect}; -use runvil::model::{Completion, CompletionResult, ModelId, ModelInfo, ThinkingMode, ToolCall}; +use runvil::model::{ + Completion, CompletionResult, ModelId, ModelInfo, ModelRole, ThinkingMode, ToolCall, +}; use runvil::timestamp::Timestamp; use runvil::tools::{ToolCatalog, ToolId, ToolInfo, ToolOutput}; -use runvil::{Prompt, Run, RunContext, RunResult, RunSeed, Step}; +use runvil::{Prompt, Run, RunConfig, RunOutcome, RunSeed, Step}; use serde_json::json; use super::*; @@ -45,8 +48,8 @@ fn tool_round(offered: &[Offered]) -> Round { } let source = format!( "---\nname: round\ndescription: One tool round.\nrunvil: 0\n\ - models:\n writer: {{}}\n---\n\n# Round\n\n## Ask\n\n\ - ```lua\nmodels.use('writer')\n{offers}messages:user('echo once')\n\ + ---\n\n# Round\n\n## Ask\n\n\ + ```lua\n{offers}messages:user('echo once')\n\ models.loop()\nreturn messages[#messages].content\n```\n" ); let first_wire_name = format!("example_test_{}", offered[0].0); @@ -65,16 +68,17 @@ fn tool_round(offered: &[Offered]) -> Round { NonZeroU32::new(8_192).expect("8192 is non-zero"), ThinkingMode::Never, ); - let ctx = RunContext::new("round", RunSeed::from_bytes([7; 32]), Timestamp::UNIX_EPOCH); + let ctx = RunConfig::new("round", RunSeed::from_bytes([7; 32]), Timestamp::UNIX_EPOCH); let catalog = ToolCatalog::new(tools).expect("the offered tools are distinct"); - let (ctx, requirements) = ctx.with_tools(catalog).prepare(&prompt, Some(model)); - assert!(requirements.refusal().is_none(), "the round prepares"); + let ctx = ctx + .with_tools(catalog) + .with_models(BTreeMap::from([(ModelRole::Default, model)])); let mut run = Run::new(Arc::new(prompt), "", ctx); let mut rounds = Vec::new(); - let result = loop { + let outcome = loop { let effects = match run.step() { Step::Pending { effects, .. } => effects, - Step::Done { result, .. } => break result, + Step::Done { outcome, .. } => break outcome, }; for IssuedEffect { id, effect, .. } in effects { let answer = match effect { @@ -112,8 +116,8 @@ fn tool_round(offered: &[Offered]) -> Round { } }; assert!( - matches!(&result, RunResult::Completed(text) if text == "done"), - "the round runs to its terminal reply: {result:?}" + matches!(&outcome, RunOutcome::Completed { final_text } if final_text == "done"), + "the round runs to its terminal reply: {outcome:?}" ); assert_eq!(rounds.len(), 2, "one tool round, then the terminal round"); rounds.pop().expect("the terminal round was recorded") diff --git a/crates/harness-gateway-client/src/wire/stream-tests.rs b/crates/harness-gateway-client/src/wire/stream-tests.rs index 2bb244b36..dd6d136a1 100644 --- a/crates/harness-gateway-client/src/wire/stream-tests.rs +++ b/crates/harness-gateway-client/src/wire/stream-tests.rs @@ -209,7 +209,6 @@ fn finish_fails_a_tool_call_batch_sharing_an_id_as_a_retryable_malformed_reply() .finish(Value::Null, None) .expect_err("two calls sharing an id must fail"); assert_eq!(error.kind(), CompletionErrorKind::MalformedResponse); - assert!(error.is_retryable()); } #[test] @@ -385,7 +384,6 @@ fn error_envelope_naming_a_context_limit_is_a_context_overflow() { .expect_err("an error envelope must fail the stream"); assert_eq!(error.kind(), CompletionErrorKind::ContextOverflow); assert_eq!(error.overflow(), (Some(5000), Some(4096))); - assert!(!error.is_retryable()); } #[test] diff --git a/crates/harness-internal/runner/src/effect_loop-flight.rs b/crates/harness-internal/runner/src/effect_loop-flight.rs index 0b2dd621f..8d78398ad 100644 --- a/crates/harness-internal/runner/src/effect_loop-flight.rs +++ b/crates/harness-internal/runner/src/effect_loop-flight.rs @@ -122,6 +122,14 @@ impl Flights { self.effects.len() + self.waiting.len() } + /// Tears down every future the set holds, aborted ones not yet polled + /// included, and every waiting call, with no answer for any of them. + pub(super) fn clear(&mut self) { + self.futures.clear(); + self.effects.clear(); + self.waiting.clear(); + } + /// Puts `answer`, the future of chat, render, or timer effect `id`, in /// flight under `provenance`. It never waits and never counts toward /// the cap. diff --git a/crates/harness-internal/runner/src/effect_loop-outcome.rs b/crates/harness-internal/runner/src/effect_loop-outcome.rs deleted file mode 100644 index 1774f19bd..000000000 --- a/crates/harness-internal/runner/src/effect_loop-outcome.rs +++ /dev/null @@ -1,35 +0,0 @@ -//! The recorded outcome of a run: what the loop hands the recorder when a -//! run ends, and what a refused preparation records in its place. - -use runvil::{RunError, RunResult}; - -use crate::display_chain::display_chain; -use crate::recorder::RunOutcome; - -/// The recorded outcome for the run's result. An outcome kind this Harness -/// does not know is recorded as an internal failure naming it, since the -/// result enum is `#[non_exhaustive]`. -pub(super) fn outcome_of(result: RunResult) -> RunOutcome { - match result { - RunResult::Completed(final_text) => RunOutcome::Completed { final_text }, - RunResult::Cancelled => RunOutcome::Cancelled, - RunResult::Failure(error) => failed_outcome(&error), - unknown => RunOutcome::Failed { - kind: "Internal".to_owned(), - message: format!( - "the run ended with an outcome this Harness does not know: {unknown:?}" - ), - }, - } -} - -/// The failed outcome for an Engine error: the failure's kind is the -/// error kind's debug name and its message the error's text with its -/// cause chain. The one derivation for a run that failed under the loop -/// and a run preparation refused, so the two agree in the record. -pub(crate) fn failed_outcome(error: &RunError) -> RunOutcome { - RunOutcome::Failed { - kind: format!("{:?}", error.kind()), - message: display_chain(error), - } -} diff --git a/crates/harness-internal/runner/src/effect_loop.rs b/crates/harness-internal/runner/src/effect_loop.rs index c7541d875..c9663e1e3 100644 --- a/crates/harness-internal/runner/src/effect_loop.rs +++ b/crates/harness-internal/runner/src/effect_loop.rs @@ -4,7 +4,7 @@ //! resume`. Every step's events are handed to the run's recorder before any //! of the step's effects is issued, so an effect's record never precedes the //! events its step reported. Each effect is appended as its -//! [`EffectRecord`](runvil::effect::EffectRecord) and then handled by +//! [`record`](runvil::effect::Effect::record) and then handled by //! kind. A chat, render, tool-call, or timer effect's performer future //! joins the effects in flight, which the loop polls inside the run's own //! future: the loop starts no task, so a run is one future that any @@ -17,30 +17,39 @@ //! inline. Real-disk operations therefore block whatever polls the run, and //! Vfs operations from different chains run one at a time in issue order. //! +//! An effect the step withdraws is answered `Dropped` without being +//! performed, and a withdrawn effect an earlier step issued is aborted +//! and answered `Dropped`, so the step that decides the run leaves the +//! loop nothing to perform. +//! //! The loop waits on three things at once: the run's cancel flag, the stop //! a Host raises through [`RunControl::stop_round`](crate::RunControl::stop_round), -//! and the next effect to land. A cancel cancels the run, aborts every -//! effect in flight, answers each `Dropped`, and steps the run to `Done`. A -//! stop aborts every effect in flight except tool calls that survive stops -//! and answers each `Dropped`, leaving the cancel flag clear, so the run -//! decides what a dropped call means: a `pcall` catches it, and an uncaught -//! one ends the run cancelled. The loop also looks for a stop before it -//! starts a step's effects, and lowers each stop it sees once it has -//! dropped what the stop reached, none included, so a stop never reaches -//! an effect started after the loop saw it. A drop made there resumes the -//! run as an inline answer does, so once the step's effects start the -//! loop steps again instead of awaiting an answer. A Vfs effect is never in -//! flight at a cancel or a stop: it is answered before the loop waits -//! again. A drop is an answer, recorded like any other, so every effect -//! record has exactly one answer record. Tool calls beyond the performer's -//! cap wait, oldest first, and a cancel, a stop, or a withdrawal reaches them. +//! and the next effect to land. A cancel aborts every effect in flight, +//! answers each `Dropped`, and steps the run, which reads the same flag, +//! to `Done`. A stop aborts every effect in flight except tool calls that +//! survive stops and answers each `Dropped`, leaving the cancel flag +//! clear, so the run decides what a dropped call means: a `pcall` catches +//! it, and an uncaught one ends the run cancelled. The loop also looks for +//! a stop before it starts a step's effects, and lowers each stop it sees +//! once it has dropped what the stop reached, none included, so a stop +//! never reaches an effect started after the loop saw it. A drop made +//! there resumes the run as an inline answer does, so once the step's +//! effects start the loop steps again instead of awaiting an answer. A Vfs +//! effect is never in flight at a cancel or a stop: it is answered before +//! the loop waits again. A drop is an answer, recorded like any other, so +//! every effect record has exactly one answer record. Tool calls beyond the +//! performer's cap wait, oldest first, and a cancel, a stop, or a +//! withdrawal reaches them. //! //! A performer that panics is caught where its future is polled: its //! effect is answered `Dropped` and the panic is logged, so a lost //! performer can never leave the run waiting forever. A Vfs operation that //! panics is caught where it runs and answered `Dropped` the same way. //! -//! The run's `Done` ends the run with its outcome at the recorder. +//! The run's `Done` ends the run with its outcome at the recorder. A run +//! that is pending with nothing issued and nothing in flight ends there as +//! an internal failure: the run reports a stall itself, so reaching this +//! means the loop lost an effect. use std::future::{Future, poll_fn}; use std::panic::{AssertUnwindSafe, catch_unwind}; @@ -57,52 +66,43 @@ use runvil::{Run, Step}; use crate::harness::StopSignal; use crate::performers::{BoxFuture, Performers}; -use crate::recorder::{Record, RecordKind, RecorderError, RunId, RunOutcome, RunRecorder}; +use crate::recorder::{ + Record, RecordKind, RecorderError, RunErrorKind, RunId, RunOutcome, RunRecorder, +}; #[path = "effect_loop-flight.rs"] mod flight; -#[path = "effect_loop-outcome.rs"] -mod outcome; use flight::{Flights, Reach}; -pub(crate) use outcome::failed_outcome; -/// Why the loop stopped without an outcome. -#[derive(Debug, thiserror::Error)] -#[non_exhaustive] -pub enum DriveError { - /// The recorder refused a write; the run cannot be recorded, so it is - /// not driven further. - #[error(transparent)] - Recorder(#[from] RecorderError), - /// The run is pending with nothing issued and nothing in flight: the - /// run reports a stall itself, so reaching this means the loop lost an - /// effect. - #[error("the effect loop has nothing to await for a pending run")] - Stalled, -} +/// The message of the failure a stalled run ends with. +const STALLED: &str = "the effect loop has nothing to await for a pending run"; /// Drives `run` to its end inside the returned future: performs its /// effects through `performers`, hands every event, effect, and answer to -/// `recorder` under `run_id`, and cancels the run when `cancel` fires. -/// Ends the run at the recorder with its outcome and returns it. +/// `recorder` under `run_id`, and drops every effect in flight when +/// `cancel` fires. `cancel` must be the flag the run's config was built +/// with, so the run sees the same cancel. Ends the run at the recorder with +/// its one outcome and returns it. +/// +/// A run that is pending with nothing to await ends as failed with the +/// kind `Internal`, once every future the loop still holds is torn down. /// /// The future is boxed internally: the step machinery is large, and the /// caller's own future stays small. It is `Send`, and it needs no runtime /// of its own: every performer future is polled inside it. /// /// # Errors -/// Returns [`DriveError::Recorder`] when the recorder refuses a write and -/// [`DriveError::Stalled`] when the run pends with nothing to await. In -/// either case every effect still in flight is torn down with the future, -/// the run is abandoned mid-flight, and the recorder is not told it ended. +/// Returns the recorder's error when it refuses a write. That is the only +/// error: every effect still in flight is torn down with the future, the +/// run is abandoned mid-flight, and the recorder is not told it ended. pub async fn drive_run( run: Run, performers: Performers, recorder: Arc, run_id: RunId, cancel: CancelHandle, -) -> Result { +) -> Result { drive(run, performers, recorder, run_id, cancel, Arc::default()).await } @@ -115,7 +115,7 @@ pub(crate) async fn drive( run_id: RunId, cancel: CancelHandle, stop: Arc, -) -> Result { +) -> Result { let cap = performers.tool.max_in_flight(); let mut driver = Driver { run, @@ -154,32 +154,25 @@ enum Woken { } impl Driver { - async fn drive(&mut self) -> Result { + async fn drive(&mut self) -> Result { loop { match self.run.step() { - Step::Done { result, events } => { + Step::Done { outcome, events } => { self.commit_events(events).await?; // `Done` is returned only once every effect is // answered, so nothing is in flight. - let outcome = outcome::outcome_of(result); - self.recorder.end_run(self.run_id, outcome.clone()).await?; - return Ok(outcome); + return self.end(outcome).await; } Step::Pending { effects, events, withdrawn, } => { - // The run's own word, not a scan of its events: the - // events are a report, not a control channel. A cancel - // that fired before this step is handed to the run here - // so its next step observes it. - if self.cancel.is_cancelled() { - self.run.cancel(); - } - let decided = self.run.decided() || self.cancel.is_cancelled(); + // The run reads this same flag, so its next step tears + // every chain down. + let cancelled = self.cancel.is_cancelled(); self.commit_events(events).await?; - if decided { + if cancelled { // Every effect issued in this step is moot before // it is performed, and every effect in flight is // moot too. Record each effect and answer it @@ -197,18 +190,26 @@ impl Driver { } for issued in effects { self.commit_effect(&issued).await?; + let (id, provenance) = (issued.id, &issued.provenance); + // The run's own word, not a scan of its events: the + // events are a report, not a control channel. A + // withdrawn effect is moot before it is performed, + // a Vfs effect included. + if withdrawn.binary_search(&id).is_ok() { + self.drop_effect(id, provenance).await?; + answered = true; + continue; + } // A Vfs effect comes back from `perform` and is // answered here; every other kind goes in flight. // The run is not stepped between inline answers: // the whole batch is resumed, then stepped once. - let (id, provenance) = (issued.id, &issued.provenance); if let Some((access, op)) = self.perform(id, provenance, issued.effect) { let answer = answer_vfs(id, provenance, access, op); self.answer(id, provenance, answer).await?; answered = true; } } - // A step can issue a call and withdraw it, so withdrawal follows queuing. if !withdrawn.is_empty() { self.drop_in_flight(Reach::Withdrawn(&withdrawn)).await?; answered = true; @@ -222,7 +223,15 @@ impl Driver { continue; } if self.flights.is_empty() { - return Err(DriveError::Stalled); + // Nothing is owed an answer, but the set may still + // hold aborted futures it has not polled. + self.flights.clear(); + return self + .end(RunOutcome::Failed { + kind: RunErrorKind::Internal, + message: STALLED.to_owned(), + }) + .await; } self.await_answer().await?; } @@ -230,15 +239,18 @@ impl Driver { } } + /// Ends the run at the recorder with `outcome` and returns it. + async fn end(&mut self, outcome: RunOutcome) -> Result { + self.recorder.end_run(self.run_id, outcome.clone()).await?; + Ok(outcome) + } + /// Waits for the next answer, then answers every effect already /// landed behind it, or acts on a cancel or a stop as soon as one is /// raised. async fn await_answer(&mut self) -> Result<(), RecorderError> { match self.wake().await { - Woken::Cancel => { - self.run.cancel(); - self.drop_in_flight(Reach::All).await - } + Woken::Cancel => self.drop_in_flight(Reach::All).await, Woken::Stop => self.act_on_stop().await.map(drop), Woken::Landed(id, provenance, answer) => { self.answer(id, &provenance, answer).await?; @@ -408,12 +420,11 @@ impl Driver { /// Appends one issued effect's record. async fn commit_effect(&mut self, issued: &IssuedEffect) -> Result<(), RecorderError> { - let payload = serde_json::to_value(issued.effect.record()).map_err(RecorderError::new)?; self.append(record( &issued.provenance, RecordKind::Effect, Some(issued.id.get()), - payload, + issued.effect.record(), )) .await } @@ -425,12 +436,11 @@ impl Driver { provenance: &Provenance, answer: &EffectAnswer, ) -> Result<(), RecorderError> { - let payload = serde_json::to_value(answer.record()).map_err(RecorderError::new)?; self.append(record( provenance, RecordKind::Answer, Some(id.get()), - payload, + answer.record(), )) .await } diff --git a/crates/harness-internal/runner/src/environment.rs b/crates/harness-internal/runner/src/environment.rs index 3543aa27c..bfe7df895 100644 --- a/crates/harness-internal/runner/src/environment.rs +++ b/crates/harness-internal/runner/src/environment.rs @@ -1,6 +1,6 @@ //! A run's environment: the Host snapshot a request carries and the //! resolution of the Host's selected model, through its inference broker, -//! into the run's context. +//! into the run's config. //! //! Everything here arrives as data or through the broker. The Harness //! never resolves a gateway, reads a menu, or names a workspace crate: the @@ -12,6 +12,7 @@ use std::path::PathBuf; use runvil::model::{CompletionError, ModelId, ModelInfo}; use crate::performers::InferenceBroker; +use crate::recorder::RunErrorKind; /// The Host state a run reads when it launches. /// @@ -43,8 +44,8 @@ impl HostSnapshot { /// The reason a run failed to resolve its model at launch. /// -/// Each variant becomes the run's error, ends the launch, and is reported -/// to the operator. +/// Each variant ends the recorded run as failed, with the error's cause +/// chain as the message. #[derive(Debug, thiserror::Error)] #[non_exhaustive] pub enum CurrentModelError { @@ -57,7 +58,19 @@ pub enum CurrentModelError { SelectionAbsent(String), } -/// Resolves the Host's current model for one run's context: lists +impl CurrentModelError { + /// The Engine's kind for the run this failure ends: a model list the + /// broker cannot fetch is a completion failure, and a selection the + /// list lacks is an unmet requirement. + pub(crate) fn kind(&self) -> RunErrorKind { + match self { + CurrentModelError::CatalogFetchFailed(_) => RunErrorKind::Completion, + CurrentModelError::SelectionAbsent(_) => RunErrorKind::RequirementsUnmet, + } + } +} + +/// Resolves the Host's current model for one run's config: lists /// `broker`'s models and looks up `host`'s selection in the list. A launch /// with no selection binds the first model the broker lists. /// diff --git a/crates/harness-internal/runner/src/files.rs b/crates/harness-internal/runner/src/files.rs index 55e2ac768..4923c85d5 100644 --- a/crates/harness-internal/runner/src/files.rs +++ b/crates/harness-internal/runner/src/files.rs @@ -10,7 +10,7 @@ use runvil::vfs::{Origin, VfsError, VfsOp, VfsOutcome, VfsRef, perform_vfs_op}; -use crate::recorder::RunOutcome; +use crate::recorder::{RunErrorKind, RunOutcome}; /// Why a run's report omits its output text. /// @@ -73,6 +73,20 @@ pub enum InputFileError { }, } +impl InputFileError { + /// The Engine's kind for the run this failure ends: input text the + /// prompt does not declare and a declared input nobody supplied are + /// unmet requirements, and a store refusal is a store failure. + pub(crate) fn kind(&self) -> RunErrorKind { + match self { + InputFileError::Undeclared | InputFileError::Missing { .. } => { + RunErrorKind::RequirementsUnmet + } + InputFileError::Vfs { .. } => RunErrorKind::Vfs, + } + } +} + /// Puts the prompt's declared input in place in `vfs`'s store: writes /// `text` at `declared`, overwriting any file there, or, when the launch /// supplied no text, checks that the store already holds the file. diff --git a/crates/harness-internal/runner/src/harness-control.rs b/crates/harness-internal/runner/src/harness-control.rs index af47f0eee..c8d06043f 100644 --- a/crates/harness-internal/runner/src/harness-control.rs +++ b/crates/harness-internal/runner/src/harness-control.rs @@ -62,9 +62,9 @@ impl RunControl { /// Every effect in flight, tool calls that survive stops included, is /// answered `Dropped`, and the run ends cancelled. Every tool call /// waiting for a slot is answered `Dropped` too, unstarted. A cancel - /// raised before the run begins ends the run before it reaches the - /// recorder. Calling `cancel` more than once is the same as calling it - /// once. + /// raised before the run begins still gives the run a record, which + /// ends cancelled before the prompt is parsed. Calling `cancel` more + /// than once is the same as calling it once. pub fn cancel(&self) { self.cancel.cancel(); } diff --git a/crates/harness-internal/runner/src/harness.rs b/crates/harness-internal/runner/src/harness.rs index f4f862304..f3eff245a 100644 --- a/crates/harness-internal/runner/src/harness.rs +++ b/crates/harness-internal/runner/src/harness.rs @@ -3,12 +3,12 @@ //! //! A Host builds a [`Harness`] from its recorder, inference broker, timer, //! built Packs, and the run's services, takes the run's [`RunControl`], and -//! awaits [`Harness::run`] on whatever executor it likes. The run resolves -//! the launch model through the broker, prepares the request's source, -//! drives the run through the effect loop, and reads the declared output -//! file of a completed run. Every performer future is polled inside that -//! one future, and every VFS operation runs inline in it, so the Harness -//! starts no task and names no runtime. +//! awaits [`Harness::run`] on whatever executor it likes. The run begins +//! its record, resolves the launch model through the broker, prepares the +//! request's source, drives the run through the effect loop, and reads the +//! declared output file of a completed run. Every performer future is +//! polled inside that one future, and every VFS operation runs inline in +//! it, so the Harness starts no task and names no runtime. use std::fmt; use std::pin::pin; @@ -16,15 +16,17 @@ use std::sync::Arc; use futures_util::future::{Either, select}; use runvil::cancel::CancelHandle; +use runvil::model::ModelInfo; use runvil::vfs::VfsRef; use runvil_plugin::HostServices; -use crate::effect_loop::{DriveError, drive}; -use crate::environment::{CurrentModelError, HostSnapshot, current_model}; +use crate::display_chain::display_chain; +use crate::effect_loop::drive; +use crate::environment::{HostSnapshot, current_model}; use crate::files::{OutputError, report_output}; use crate::host::HostContext; use crate::performers::{InferenceBroker, Timer}; -use crate::prepare::{PrepareError, Services, prepare_noting_cancel}; +use crate::prepare::{Services, begin, prepare}; use crate::recorder::{RecorderError, RunId, RunOutcome, RunRecorder}; #[path = "harness-control.rs"] @@ -82,31 +84,27 @@ pub struct RunRequest { } /// How one run ended, and what it left at its declared output file. +/// +/// The recorder holds the same outcome under the same run id. #[derive(Debug, Clone, PartialEq, Eq)] pub struct RunReport { - /// The id the recorder issued for the run. `None` when a cancel ended - /// the run before the recorder began it. - pub run_id: Option, - /// How the run ended. When `run_id` is `Some`, the recorder holds this - /// same outcome. + /// The id the recorder issued for the run. + pub run_id: RunId, + /// How the run ended. pub outcome: RunOutcome, /// The text a completed run left at its declared `output:` file, or /// the reason the report omits that text. pub output: Result, } -/// An error that stopped a run before `Harness::run` could report an -/// outcome. +/// The error that stopped a run before `Harness::run` could record its +/// outcome: the recorder refused a write. /// -/// `Harness::run` returns it as its `Err` value. +/// `Harness::run` returns it as its `Err` value. Every other way a run +/// ends is a recorded outcome in a [`RunReport`]. #[derive(Debug, thiserror::Error)] #[non_exhaustive] pub enum HarnessError { - /// The Harness failed to resolve the Host's current model through the - /// inference broker. The resolution failure is the error's source. The - /// run failed to begin. - #[error("the run's model could not be resolved")] - Model(#[source] CurrentModelError), /// The recorder refused a write, so the Harness stops driving the run. #[error("the run could not be recorded")] Recorder { @@ -117,10 +115,6 @@ pub enum HarnessError { #[source] source: RecorderError, }, - /// The run was still pending, but its set of effects in flight was - /// empty. This means the Harness lost track of an effect. - #[error("the effect loop has nothing to await for a pending run")] - Stalled, } impl Harness { @@ -162,28 +156,33 @@ impl Harness { /// Runs `request` to its end and reports how it ended. /// - /// It resolves the run's model through the inference broker, prepares - /// the prompt from its source, waits for the Host's Packs to be - /// ready, drives the run, and reads the declared output file if the - /// run completed. The returned future is `Send`. Its only runtime needs - /// are those of the Host's performers and Packs. + /// It begins the run's record, resolves the run's model through the + /// inference broker, prepares the prompt from its source, waits for + /// the Host's Packs to be ready, drives the run, and reads the declared + /// output file if the run completed. The returned future is `Send`. + /// Its only runtime needs are those of the Host's performers and Packs. /// - /// Three problems end the run as failed and still return a report with - /// that outcome: a source that fails to parse, a declared input file - /// the Harness fails to put in place, and an environment that falls - /// short of the prompt's requirements. A cancel raised before calling - /// `run`, or while the broker lists its models, returns a report with - /// the outcome `Cancelled` and `run_id` set to `None`. An environment - /// that falls short after a cancel was raised, as when the cancel cuts - /// short the wait for a Pack that is still starting, ends the run as - /// `Cancelled` instead, and the report carries the run's `run_id`. + /// Every run gets a record and ends with one outcome, which the report + /// carries beside the run's id. The record begins before anything + /// else, so a run that ends before it starts is recorded too. A cancel + /// raised before calling `run`, or while the broker lists its models, + /// ends the run as `Cancelled`. A model list the broker fails to fetch + /// ends it as failed with the kind `Completion`, and a list that omits + /// the selected model ends it as failed with the kind + /// `RequirementsUnmet`. A source that fails to parse, a declared input + /// file the Harness fails to put in place, and an environment that + /// falls short of the prompt's requirements, which the run's first + /// step finds, also end it as failed. An environment that falls short + /// after a cancel was raised, as when the cancel cuts short the wait + /// for a Pack that is still starting, ends the run as `Cancelled` + /// instead. A run that is pending with nothing to await, which means + /// the Harness lost track of an effect, ends as failed with the kind + /// `Internal`. /// /// # Errors - /// Returns [`HarnessError::Model`] when the broker fails to list its - /// models or its list omits the selected one. Returns - /// [`HarnessError::Recorder`] when the recorder refuses a write. - /// Returns [`HarnessError::Stalled`] when the run is pending and its - /// set of effects in flight is empty. + /// Returns [`HarnessError::Recorder`] when the recorder refuses a + /// write. That is the only error: the run is then not recorded to its + /// end. pub async fn run(self, request: RunRequest) -> Result { Box::pin(self.run_to_end(request)).await } @@ -206,19 +205,27 @@ impl Harness { host, } = request; - let model = { - let resolving = pin!(current_model(&host, &*broker)); - // The cancel arm is polled first, so a cancel raised before the - // run, or before a listing that is ready at once, wins. - match select(control.cancel_handle().cancelled(), resolving).await { - Either::Left(((), _)) => { - return Ok(RunReport { - run_id: None, - outcome: RunOutcome::Cancelled, - output: Err(OutputError::NotCompleted), - }); - } - Either::Right((resolved, _)) => resolved.map_err(HarnessError::Model)?, + let begun = begin(&*recorder, &name, &source) + .await + .map_err(|source| HarnessError::Recorder { run: None, source })?; + let run_id = begun.run_id; + let recorded = |source| HarnessError::Recorder { + run: Some(run_id), + source, + }; + + let model = match resolve_model(&host, &*broker, control.cancel_handle()).await { + Ok(model) => model, + Err(outcome) => { + recorder + .end_run(run_id, outcome.clone()) + .await + .map_err(recorded)?; + return Ok(RunReport { + run_id, + outcome, + output: Err(OutputError::NotCompleted), + }); } }; @@ -235,28 +242,20 @@ impl Harness { model, ui: Some(host.ui()), }; - let prepared = match prepare_noting_cancel(&source, &args, services).await { + let prepared = match prepare(&source, &args, begun, services).await { Ok(prepared) => prepared, - Err((PrepareError::Refused { run_id, .. }, true)) => { - return Ok(RunReport { - run_id: Some(run_id), - outcome: RunOutcome::Cancelled, - output: Err(OutputError::NotCompleted), - }); - } - Err((error, _)) => { + Err(error) => { return match error.ended() { - Ok((run_id, outcome)) => Ok(RunReport { - run_id: Some(run_id), + Ok(outcome) => Ok(RunReport { + run_id, outcome, output: Err(OutputError::NotCompleted), }), - Err((run, source)) => Err(HarnessError::Recorder { run, source }), + Err(source) => Err(recorded(source)), }; } }; - let run_id = prepared.run_id; let outcome = drive( prepared.run, prepared.performers, @@ -266,18 +265,33 @@ impl Harness { Arc::clone(control.stop_signal()), ) .await - .map_err(|error| match error { - DriveError::Recorder(source) => HarnessError::Recorder { - run: Some(run_id), - source, - }, - DriveError::Stalled => HarnessError::Stalled, - })?; + .map_err(recorded)?; let output = report_output(&vfs, prepared.output_path.as_deref(), &outcome); Ok(RunReport { - run_id: Some(run_id), + run_id, outcome, output, }) } } + +/// Resolves the Host's current model through `broker`, or returns the +/// outcome that ends the run instead: `Cancelled` when `cancel` fires +/// first, and a failure with the lookup's kind and cause chain when the +/// broker's list cannot be fetched or lacks the selection. +async fn resolve_model( + host: &HostSnapshot, + broker: &dyn InferenceBroker, + cancel: &CancelHandle, +) -> Result, RunOutcome> { + let resolving = pin!(current_model(host, broker)); + // The cancel arm is polled first, so a cancel raised before the run, + // or before a listing that is ready at once, wins. + match select(cancel.cancelled(), resolving).await { + Either::Left(((), _)) => Err(RunOutcome::Cancelled), + Either::Right((resolved, _)) => resolved.map_err(|error| RunOutcome::Failed { + kind: error.kind(), + message: display_chain(&error), + }), + } +} diff --git a/crates/harness-internal/runner/src/host-run.rs b/crates/harness-internal/runner/src/host-run.rs index 37a57209f..5c6c05454 100644 --- a/crates/harness-internal/runner/src/host-run.rs +++ b/crates/harness-internal/runner/src/host-run.rs @@ -16,12 +16,12 @@ use std::collections::{BTreeMap, BTreeSet}; use std::fmt; use std::sync::Arc; +use runvil::Prompt; use runvil::effect::{PackToolSets, ToolCallOrigin}; use runvil::model::ModelInfo; -use runvil::packs::Prelude; +use runvil::packs::{Prelude, RosterEntry}; use runvil::tools::{ToolCatalog, ToolError, ToolErrorKind, ToolId, ToolInfo, ToolOutput}; use runvil::vfs::Access; -use runvil::{MissingService, Prompt, Requirements, UnavailablePack, UsablePack}; use runvil_plugin::{HostServices, Pack, PackId, PackInfo, RenderContext, ServiceId, ToolContext}; use serde_json::Value; @@ -39,8 +39,8 @@ pub(crate) struct HostRunContext { enum PackState { /// It can serve; its validated tools are in the run's catalog. Usable { info: PackInfo, pack: Arc }, - /// Its `construct` failed, or its `ready` did before this run, for - /// this reason. + /// The pick could not resolve it, or its `construct` failed, or its + /// `ready` did before this run, for this reason. Unavailable(String), /// The run lacks these services from the Pack's needs. NeedsUnmet(Vec), @@ -54,18 +54,22 @@ impl HostRunContext { /// reason it maps to. Any other built Pack is usable when `services` /// provide every id the Pack needs; only then is its tool list read /// and validated. The run's one catalog is built from every usable - /// Pack's validated tools, in Pack id order. + /// Pack's validated tools, in Pack id order. Each declared Pack id the + /// pick could not resolve that `unresolved` gives a reason for is + /// unavailable with that reason; one with no reason stays out, so the + /// run finds it missing. pub(super) fn snapshot<'a>( packs: impl Iterator, String>)>, services: HostServices, mut failures: BTreeMap<&'static str, String>, + unresolved: &BTreeMap>, ) -> HostRunContext { let mut seen = BTreeSet::new(); // The input arrives in pick order, declared Packs first, so the // catalog is built after `packs` is collected, from these lists // in Pack id order. let mut usable_tools: BTreeMap<&PackId, Vec> = BTreeMap::new(); - let packs = packs + let mut packs: BTreeMap = packs .map(|(name, info, built)| { let state = match (built, failures.remove(info.fqn)) { (Err(reason), _) => PackState::Unavailable(reason.clone()), @@ -91,6 +95,10 @@ impl HostRunContext { (name.clone(), state) }) .collect(); + packs.extend(unresolved.iter().filter_map(|(name, reason)| { + let reason = reason.clone()?; + Some((name.clone(), PackState::Unavailable(reason))) + })); let catalog = ToolCatalog::new(usable_tools.into_values().flatten()).unwrap_or_else(|error| { // Every tool passed containment and uniqueness validation one @@ -110,35 +118,40 @@ impl HostRunContext { self.catalog.clone() } - /// The run's roster: each of its usable Packs, with or without tools, - /// with the vendor of its FQN, and an entry with the reason for each - /// one that can't serve, both in id order. - pub(crate) fn roster(&self) -> (Vec, Vec) { - let mut usable = Vec::new(); - let mut unavailable = Vec::new(); - for (name, pack) in &self.packs { - match pack { - PackState::Usable { info, .. } => { - let vendor = info - .fqn - .split_once('/') - .map_or(info.fqn, |(vendor, _)| vendor); - usable.push(UsablePack::new(name.clone(), vendor)); - } - PackState::Unavailable(reason) => { - unavailable.push(UnavailablePack::new(name.clone(), reason.clone())); - } - PackState::NeedsUnmet(unmet) => { - let services: Vec = unmet.iter().map(ToString::to_string).collect(); - let reason = format!( - "the run lacks the services it needs: {}", - services.join(", ") - ); - unavailable.push(UnavailablePack::new(name.clone(), reason)); - } - } - } - (usable, unavailable) + /// The run's roster, the one account of how each of its Packs stands: + /// each usable Pack, with or without tools, with the vendor of its + /// FQN, and each one that can't serve with the reason, a declared Pack + /// the pick could not resolve included, in id order. + pub(crate) fn roster(&self) -> BTreeMap { + self.packs + .iter() + .map(|(name, pack)| { + let entry = match pack { + PackState::Usable { info, .. } => { + let vendor = info + .fqn + .split_once('/') + .map_or(info.fqn, |(vendor, _)| vendor); + RosterEntry::Usable { + vendor: vendor.to_owned(), + } + } + PackState::Unavailable(reason) => RosterEntry::Unavailable { + reason: reason.clone(), + }, + PackState::NeedsUnmet(unmet) => { + let services: Vec = unmet.iter().map(ToString::to_string).collect(); + RosterEntry::Unavailable { + reason: format!( + "the run lacks the services it needs: {}", + services.join(", ") + ), + } + } + }; + (name.clone(), entry) + }) + .collect() } /// The preludes of the usable Packs `prompt` declares, in @@ -157,54 +170,6 @@ impl HostRunContext { .collect() } - /// What the run cannot meet for `prompt`, for each Pack it declares: - /// one the pick could not resolve, as `unresolved` holds it, missing - /// or unavailable with its reason, one that failed to build or to be - /// ready, and one whose needs the run lacks. - pub(super) fn requirements( - &self, - prompt: &Prompt, - unresolved: &BTreeMap>, - ) -> Requirements { - let mut requirements = Requirements::default(); - for name in prompt - .frontmatter() - .uses() - .iter() - .map(|entry| entry.id().clone()) - { - if let Some(reason) = unresolved.get(&name) { - match reason { - None => requirements.missing_packs.push(name), - Some(reason) => requirements - .unavailable_packs - .push(UnavailablePack::new(name, reason.clone())), - } - continue; - } - match self.packs.get(&name) { - None => requirements.missing_packs.push(name), - Some(PackState::Unavailable(reason)) => requirements - .unavailable_packs - .push(UnavailablePack::new(name, reason.clone())), - Some(PackState::NeedsUnmet(unmet)) => { - for service in unmet { - tracing::warn!( - pack = %name, - %service, - "the prompt needs a Pack whose service this run lacks" - ); - requirements - .missing_services - .push(MissingService::new(name.clone(), service.to_string())); - } - } - Some(PackState::Usable { .. }) => {} - } - } - requirements - } - /// The usable Pack whose id is `pack`. fn usable(&self, pack: &PackId) -> Option<&Arc> { match self.packs.get(pack)? { diff --git a/crates/harness-internal/runner/src/host-tests-pick.rs b/crates/harness-internal/runner/src/host-tests-pick.rs index 9c585bd6f..9ed03c14d 100644 --- a/crates/harness-internal/runner/src/host-tests-pick.rs +++ b/crates/harness-internal/runner/src/host-tests-pick.rs @@ -1,8 +1,9 @@ //! The pick of each run's Packs: a bare entry means the runvil Pack of //! its id, a declared third-party Pack stands in for the runvil Pack of //! its id, an undeclared third-party Pack is in no run, what the pick -//! cannot resolve is reported with its reason, and the ready wait covers -//! only the run's Packs, keying each failure by FQN. +//! cannot resolve is on the roster with its reason or, with none, left +//! off it, and the ready wait covers only the run's Packs, keying each +//! failure by FQN. use futures_util::FutureExt as _; use runvil::cancel::CancelHandle; @@ -31,8 +32,8 @@ fn a_bare_entry_runs_with_the_runvil_pack_of_that_id() { catalog_ids(&host, HostServices::new(), &declares), ["web/runvil"] ); - let (_run, requirements) = begin(&host, HostServices::new(), BTreeMap::new(), &declares); - assert!(requirements.is_satisfied(), "{requirements:?}"); + let run = begin(&host, HostServices::new(), BTreeMap::new(), &declares); + assert_eq!(run.roster().get(&name("web")), Some(&usable_from("runvil"))); } #[test] @@ -64,65 +65,94 @@ fn a_run_that_declares_nothing_holds_every_runvil_pack_and_no_third_party_one() } #[test] -fn a_bare_entry_that_only_another_vendor_has_is_refused_naming_each_fqn() { +fn a_bare_entry_that_only_another_vendor_has_is_unavailable_naming_each_fqn() { let one = vendors("ready"); - let (_run, requirements) = begin( - &one, - HostServices::new(), - BTreeMap::new(), - &prompt("uses:\n - poop\n"), - ); + let declares = prompt("uses:\n - poop\n"); + let run = begin(&one, HostServices::new(), BTreeMap::new(), &declares); assert_eq!( - requirements.unavailable_packs, - [UnavailablePack::new( - name("poop"), + run.roster().get(&name("poop")), + Some(&unavailable_for( "poop means runvil/poop, which is not installed; write acme/poop to use that Pack" - )] + )) + ); + assert_eq!( + refusal(&run, &declares), + "the environment cannot satisfy this prompt:\n\ + - poop is unavailable: poop means runvil/poop, which is not installed; \ + write acme/poop to use that Pack", + "the run names the reason and does not call the Pack missing" ); - assert!(requirements.missing_packs.is_empty()); let both = host(&[ (info("zed/poop"), Value::Null), (info("acme/poop"), Value::Null), ]); - let (_run, requirements) = begin( - &both, - HostServices::new(), - BTreeMap::new(), - &prompt("uses:\n - poop\n"), - ); + let run = begin(&both, HostServices::new(), BTreeMap::new(), &declares); assert_eq!( - requirements.unavailable_packs, - [UnavailablePack::new( - name("poop"), + run.roster().get(&name("poop")), + Some(&unavailable_for( "poop means runvil/poop, which is not installed; \ write acme/poop or zed/poop to use that Pack" - )] + )) ); } #[test] fn an_entry_no_vendor_has_is_missing_and_an_absent_vendor_name_is_unavailable() { let host = vendors("ready"); - let (_run, requirements) = begin( - &host, - HostServices::new(), - BTreeMap::new(), - &prompt("uses:\n - nope\n - acme/gone\n"), + let declares = prompt("uses:\n - nope\n - acme/gone\n"); + let run = begin(&host, HostServices::new(), BTreeMap::new(), &declares); + let roster = run.roster(); + assert_eq!( + roster.get(&name("nope")), + None, + "no reason, so off the roster" ); - assert_eq!(requirements.missing_packs, [name("nope")]); assert_eq!( - requirements.unavailable_packs, - [UnavailablePack::new( - name("gone"), - "acme/gone is not installed" - )] - ); - let notice = requirements.notice(); - assert!(notice.contains("- missing required Pack: nope"), "{notice}"); - assert!( - notice.contains("- gone is unavailable: acme/gone is not installed"), - "{notice}" + roster.get(&name("gone")), + Some(&unavailable_for("acme/gone is not installed")) + ); + assert_eq!( + refusal(&run, &declares), + "the environment cannot satisfy this prompt:\n\ + - missing required Pack: nope\n\ + - gone is unavailable: acme/gone is not installed" + ); +} + +#[test] +fn the_roster_covers_every_declared_pack_the_run_cannot_serve() { + let needy = info("acme/needy").needs(NEEDS_SESSION); + let broken = PackInfo::new("acme/broken", backed); + let host = host(&[ + (needy, Value::Null), + (broken, Value::Null), + (info("acme/fine"), Value::Null), + ]); + let declares = prompt("uses:\n - acme/needy\n - absent\n - acme/fine\n - acme/broken\n"); + let run = begin(&host, HostServices::new(), BTreeMap::new(), &declares); + assert_eq!( + run.roster(), + BTreeMap::from([ + ( + name("broken"), + unavailable_for("backed needs tests/backend, and this host provides none"), + ), + (name("fine"), usable_from("acme")), + ( + name("needy"), + unavailable_for("the run lacks the services it needs: tests/session"), + ), + (name("subagents"), usable_from("runvil")), + ]), + "the absent Pack, which no vendor has, stays off the roster" + ); + assert_eq!( + refusal(&run, &declares), + "the environment cannot satisfy this prompt:\n\ + - missing required Pack: absent\n\ + - needy is unavailable: the run lacks the services it needs: tests/session\n\ + - broken is unavailable: backed needs tests/backend, and this host provides none" ); } @@ -161,20 +191,18 @@ fn a_ready_failure_is_keyed_by_fqn_and_never_reaches_another_vendors_pack() { failures, BTreeMap::from([("acme/web", "web will not be ready".to_owned())]) ); - let (_run, requirements) = - host.begin_run(&packs, HostServices::new(), failures, &declares_acme); + let run = host.begin_run(&packs, HostServices::new(), failures); assert_eq!( - requirements.unavailable_packs, - [UnavailablePack::new(name("web"), "web will not be ready")] + run.roster().get(&name("web")), + Some(&unavailable_for("web will not be ready")) ); let declares_runvil = prompt("uses:\n - web\n"); let packs = host.pick(&declares_runvil); let failures = host.wait_ready(&packs, &cancel).now_or_never().unwrap(); assert!(failures.is_empty(), "{failures:?}"); - let (run, requirements) = - host.begin_run(&packs, HostServices::new(), failures, &declares_runvil); - assert!(requirements.is_satisfied(), "{requirements:?}"); + let run = host.begin_run(&packs, HostServices::new(), failures); + assert_eq!(run.roster().get(&name("web")), Some(&usable_from("runvil"))); let ids: Vec = run .catalog() .tools() diff --git a/crates/harness-internal/runner/src/host-tests-render.rs b/crates/harness-internal/runner/src/host-tests-render.rs index 4d67c4913..1002a3909 100644 --- a/crates/harness-internal/runner/src/host-tests-render.rs +++ b/crates/harness-internal/runner/src/host-tests-render.rs @@ -130,7 +130,7 @@ async fn a_render_asks_each_usable_pack_once_per_set_it_was_offered_into_and_ans (needy, offering(&["needy/a"])), ], ); - let (run, _requirements) = begin(&host, HostServices::new(), BTreeMap::new(), &prompt("")); + let run = begin(&host, HostServices::new(), BTreeMap::new(), &prompt("")); let offered = vec![ sets("two", &["two/a"], &[]), sets("one", &["one/a"], &["one/b"]), @@ -173,7 +173,7 @@ async fn a_render_asks_each_usable_pack_once_per_set_it_was_offered_into_and_ans #[tokio::test] async fn a_call_goes_to_the_pack_its_id_names_when_the_catalog_lacks_the_tool() { let host = host(&[(info("runvil/one"), offering(&["one/echo"]))]); - let (run, _requirements) = begin(&host, HostServices::new(), BTreeMap::new(), &prompt("")); + let run = begin(&host, HostServices::new(), BTreeMap::new(), &prompt("")); let extra = ToolId::parse("one/extra").unwrap(); assert!( run.catalog().get(&extra).is_none(), diff --git a/crates/harness-internal/runner/src/host-tests.rs b/crates/harness-internal/runner/src/host-tests.rs index be86d9c98..381f41188 100644 --- a/crates/harness-internal/runner/src/host-tests.rs +++ b/crates/harness-internal/runner/src/host-tests.rs @@ -1,7 +1,7 @@ //! Tests for the Host's Packs and each run's snapshot: `new` builds every //! registered Pack, the built-in included, and stores a construct failure -//! as unavailable; and the snapshot's requirements, roster, catalog, -//! preludes, and calls. The pick of each run's Packs and the ready wait +//! as unavailable; and the snapshot's roster, catalog, preludes, and +//! calls. The pick of each run's Packs and the ready wait //! over them sit in the `pick` child module, and the snapshot's renders //! and its routing of a call the catalog lacks in the `render` child //! module. @@ -10,8 +10,10 @@ use std::collections::BTreeMap; use std::sync::{Arc, Mutex}; use runvil::effect::{ToolCallOrigin, ToolCaller}; +use runvil::packs::RosterEntry; +use runvil::timestamp::Timestamp; use runvil::vfs::{Origin, VfsRef}; -use runvil::{MissingService, Prompt, Requirements, UnavailablePack, UsablePack}; +use runvil::{Prompt, Run, RunConfig, RunOutcome, RunSeed, Step}; use runvil_plugin::{ HostServices, Pack, PackFuture, PackId, PackInfo, PackRegistry, ServiceId, ServiceKey, ToolContext, ToolError, ToolId, ToolInfo, ToolOutput, @@ -183,14 +185,14 @@ fn begin( services: HostServices, failures: BTreeMap<&'static str, String>, prompt: &Prompt, -) -> (HostRunContext, Requirements) { +) -> HostRunContext { let packs = host.pick(prompt); - host.begin_run(&packs, services, failures, prompt) + host.begin_run(&packs, services, failures) } /// The ids in a run's catalog, in catalog order. fn catalog_ids(host: &HostContext, services: HostServices, prompt: &Prompt) -> Vec { - let (run, _requirements) = begin(host, services, BTreeMap::new(), prompt); + let run = begin(host, services, BTreeMap::new(), prompt); run.catalog() .tools() .iter() @@ -200,9 +202,44 @@ fn catalog_ids(host: &HostContext, services: HostServices, prompt: &Prompt) -> V /// The run's usable Pack ids, in id order. fn usable(host: &HostContext, prompt: &Prompt) -> Vec { - let (run, _requirements) = begin(host, HostServices::new(), BTreeMap::new(), prompt); - let (usable, _unavailable) = run.roster(); - usable.iter().map(|entry| entry.pack.to_string()).collect() + let run = begin(host, HostServices::new(), BTreeMap::new(), prompt); + run.roster() + .into_iter() + .filter(|(_, entry)| matches!(entry, RosterEntry::Usable { .. })) + .map(|(name, _)| name.to_string()) + .collect() +} + +/// The roster entry of a Pack usable from `vendor`. +fn usable_from(vendor: &str) -> RosterEntry { + RosterEntry::Usable { + vendor: vendor.to_owned(), + } +} + +/// The roster entry of a Pack unavailable for `reason`. +fn unavailable_for(reason: &str) -> RosterEntry { + RosterEntry::Unavailable { + reason: reason.to_owned(), + } +} + +/// The refusal notice the first step of a run of `prompt` over `run`'s +/// roster fails with. +fn refusal(run: &HostRunContext, prompt: &Prompt) -> String { + let ctx = RunConfig::new( + "host-test", + RunSeed::from_bytes([0; 32]), + Timestamp::UNIX_EPOCH, + ) + .with_packs(run.roster()); + match Run::new(Arc::new(prompt.clone()), "", ctx).step() { + Step::Done { + outcome: RunOutcome::Failed { message, .. }, + .. + } => message, + other => panic!("the run's first step refuses it: {other:?}"), + } } #[test] @@ -213,20 +250,19 @@ fn the_roster_hands_each_usable_pack_with_the_vendor_the_run_picked() { (info("runvil/web"), offering(&["web/fetch"])), (info("acme/poop"), Value::Null), ]); - let (run, _requirements) = begin( + let run = begin( &host, HostServices::new(), BTreeMap::new(), &prompt("uses:\n - acme/kit\n"), ); - let (usable, _unavailable) = run.roster(); assert_eq!( - usable, - [ - UsablePack::new(name("kit"), "acme"), - UsablePack::new(name("subagents"), "runvil"), - UsablePack::new(name("web"), "runvil"), - ], + run.roster(), + BTreeMap::from([ + (name("kit"), usable_from("acme")), + (name("subagents"), usable_from("runvil")), + (name("web"), usable_from("runvil")), + ]), "the declared acme/kit reports acme, and every undeclared Pack reports runvil" ); } @@ -271,75 +307,45 @@ fn a_run_over_a_fresh_registry_holds_a_usable_subagents_pack_with_no_tools() { fn a_construct_failure_is_stored_and_reported_as_unavailable_with_its_reason() { let backed_info = PackInfo::new("acme/backed", backed); let bare = host(&[(backed_info, Value::Null)]); - let (_run, requirements) = begin( - &bare, - HostServices::new(), - BTreeMap::new(), - &prompt("uses:\n - acme/backed\n"), - ); + let declares = prompt("uses:\n - acme/backed\n"); + let run = begin(&bare, HostServices::new(), BTreeMap::new(), &declares); assert_eq!( - requirements.unavailable_packs, - [UnavailablePack::new( - name("backed"), + run.roster().get(&name("backed")), + Some(&unavailable_for( "backed needs tests/backend, and this host provides none" - )] + )) + ); + assert_eq!( + refusal(&run, &declares), + "the environment cannot satisfy this prompt:\n\ + - backed is unavailable: backed needs tests/backend, and this host provides none", + "the run names the reason, and never calls the built Pack missing" ); - assert!(requirements.missing_packs.is_empty()); let mut wide = HostServices::new(); wide.provide(&BACKEND, Arc::from("up")).unwrap(); let provided = host_with(wide, &[(backed_info, Value::Null)]); - let (_run, requirements) = begin( - &provided, - HostServices::new(), - BTreeMap::new(), - &prompt("uses:\n - acme/backed\n"), - ); - assert!( - requirements.is_satisfied(), - "construct reads the Host-wide services: {requirements:?}" - ); -} - -#[test] -fn begin_run_reports_every_declared_pack_it_cannot_serve() { - let needy = info("acme/needy").needs(NEEDS_SESSION); - let broken = PackInfo::new("acme/broken", backed); - let host = host(&[ - (needy, Value::Null), - (broken, Value::Null), - (info("acme/fine"), Value::Null), - ]); - - let (_run, requirements) = begin( - &host, - HostServices::new(), - BTreeMap::new(), - &prompt("uses:\n - acme/needy\n - absent\n - acme/fine\n - acme/broken\n"), - ); - assert_eq!(requirements.missing_packs, [name("absent")]); - assert_eq!( - requirements.missing_services, - [MissingService::new(name("needy"), "tests/session")] - ); + let run = begin(&provided, HostServices::new(), BTreeMap::new(), &declares); assert_eq!( - requirements.unavailable_packs.len(), - 1, - "the declared Pack that failed to build is reported: {requirements:?}" + run.roster().get(&name("backed")), + Some(&usable_from("acme")), + "construct reads the Host-wide services" ); - assert_eq!(requirements.unavailable_packs[0].pack, name("broken")); } #[test] fn a_declared_builtin_is_met_over_a_fresh_registry() { let host = host(&[]); - let (_run, requirements) = begin( + let run = begin( &host, HostServices::new(), BTreeMap::new(), &prompt("uses:\n - subagents\n"), ); - assert!(requirements.is_satisfied(), "{requirements:?}"); + assert_eq!( + run.roster().get(&name("subagents")), + Some(&usable_from("runvil")) + ); } #[test] @@ -347,7 +353,7 @@ fn the_builtin_adds_no_catalog_tool_and_no_prelude() { let host = host(&[]); let declares = prompt("uses:\n - subagents\n"); assert!(catalog_ids(&host, HostServices::new(), &declares).is_empty()); - let (run, _requirements) = begin(&host, HostServices::new(), BTreeMap::new(), &declares); + let run = begin(&host, HostServices::new(), BTreeMap::new(), &declares); assert!(run.preludes(&declares).is_empty()); } @@ -356,28 +362,24 @@ fn the_roster_lists_the_builtin_stub_as_usable_and_each_pack_that_cannot_serve_w let needy = info("acme/needy").needs(NEEDS_SESSION); let host = host(&[(info("acme/kit"), Value::Null), (needy, Value::Null)]); let failures = BTreeMap::from([("acme/kit", "the kit server is down".to_owned())]); - let (run, _requirements) = begin( + let run = begin( &host, HostServices::new(), failures, &prompt("uses:\n - acme/kit\n - acme/needy\n"), ); - let (usable, unavailable) = run.roster(); assert_eq!( - usable, - [UsablePack::new(name("subagents"), "runvil")], - "the undeclared stub with no tools is on the roster" - ); - assert_eq!( - unavailable, - [ - UnavailablePack::new(name("kit"), "the kit server is down"), - UnavailablePack::new( + run.roster(), + BTreeMap::from([ + (name("kit"), unavailable_for("the kit server is down")), + ( name("needy"), - "the run lacks the services it needs: tests/session" + unavailable_for("the run lacks the services it needs: tests/session"), ), - ], - "a failed ready and an unmet need each carry their reason" + (name("subagents"), usable_from("runvil")), + ]), + "the undeclared stub with no tools is on the roster, and a failed ready and an \ + unmet need each carry their reason" ); } @@ -417,7 +419,7 @@ fn an_undeclared_runvil_packs_tools_reach_the_catalog_but_its_prelude_does_not() catalog_ids(&host, HostServices::new(), &prompt), ["declared/a", "extra/b"] ); - let (run, _requirements) = begin(&host, HostServices::new(), BTreeMap::new(), &prompt); + let run = begin(&host, HostServices::new(), BTreeMap::new(), &prompt); let preludes: Vec = run .preludes(&prompt) .iter() @@ -435,7 +437,7 @@ fn preludes_follow_the_prompts_declaration_order_and_carry_the_pack_id() { with_prelude("runvil/third"), ]); let prompt = prompt("uses:\n - third\n - acme/second\n - first\n"); - let (run, _requirements) = begin(&host, HostServices::new(), BTreeMap::new(), &prompt); + let run = begin(&host, HostServices::new(), BTreeMap::new(), &prompt); let order: Vec = run .preludes(&prompt) .iter() @@ -465,7 +467,7 @@ async fn a_call_goes_to_the_pack_its_tool_ids_first_segment_names() { (info("runvil/one"), offering(&["one/echo"])), (info("runvil/two"), offering(&["two/echo"])), ]); - let (run, _requirements) = begin(&host, HostServices::new(), BTreeMap::new(), &prompt("")); + let run = begin(&host, HostServices::new(), BTreeMap::new(), &prompt("")); let answer = |result: Result| result.unwrap().text().to_owned(); assert_eq!(answer(call(&run, "one/echo").await), "one:one/echo"); @@ -480,7 +482,7 @@ async fn a_call_goes_to_the_pack_its_tool_ids_first_segment_names() { async fn a_call_to_a_pack_the_run_cannot_use_fails_naming_the_tool() { let needy = info("runvil/needy").needs(NEEDS_SESSION); let host = host(&[(needy, offering(&["needy/use"]))]); - let (run, _requirements) = begin(&host, HostServices::new(), BTreeMap::new(), &prompt("")); + let run = begin(&host, HostServices::new(), BTreeMap::new(), &prompt("")); let error = call(&run, "needy/use") .await .expect_err("the run lacks the Pack's needs"); diff --git a/crates/harness-internal/runner/src/host.rs b/crates/harness-internal/runner/src/host.rs index 03c52f3a8..9070ec6de 100644 --- a/crates/harness-internal/runner/src/host.rs +++ b/crates/harness-internal/runner/src/host.rs @@ -16,9 +16,10 @@ //! those Packs and waits for each of them to be ready, and a Pack whose //! `ready` fails is unavailable to that run the same way. Then it takes a //! snapshot of them with the run's own services: the catalog the Engine -//! offers the run, the preludes of the Packs the prompt declares, the -//! requirements the pick and the snapshot cannot meet, and the performer -//! that sends each tool call to the Pack its id names. +//! offers the run, the roster that says how each Pack stands, a declared +//! Pack the pick could not resolve included, the preludes of the Packs +//! the prompt declares, and the performer that sends each tool call to +//! the Pack its id names. use std::collections::BTreeMap; use std::fmt; @@ -26,8 +27,8 @@ use std::pin::pin; use std::sync::Arc; use futures_util::future::{Either, join_all, select}; +use runvil::Prompt; use runvil::cancel::CancelHandle; -use runvil::{Prompt, Requirements}; use runvil_plugin::{HostServices, Pack, PackId, PackInfo, PackRegistry}; #[path = "host-run.rs"] @@ -191,25 +192,23 @@ impl HostContext { } /// Takes one run's snapshot of its `packs` with `services`, the run's - /// own services, and reports the requirements it cannot meet for - /// `prompt`. + /// own services. /// /// `failures` holds the reason of each Pack whose `ready` failed /// before this run, by FQN, which the run treats as unavailable. The /// snapshot gives the run's catalog, which holds the tools of each of /// its usable Packs, its roster, which names each of its Packs with - /// the reason of each one that can't serve, and its declared preludes, - /// the declared, usable Packs', in declaration order. It also performs - /// the run's tool calls. The requirements cover every Pack the prompt - /// declares, the ones the pick could not resolve included. + /// the reason of each one that can't serve, a declared Pack the pick + /// could not resolve with a reason included, and its declared + /// preludes, the declared, usable Packs', in declaration order. It also + /// performs the run's tool calls. pub(crate) fn begin_run( &self, packs: &RunPacks, services: HostServices, failures: BTreeMap<&'static str, String>, - prompt: &Prompt, - ) -> (HostRunContext, Requirements) { - let run = HostRunContext::snapshot( + ) -> HostRunContext { + HostRunContext::snapshot( packs .fqns .iter() @@ -217,9 +216,8 @@ impl HostContext { .map(|built| (&built.id, built.info, &built.pack)), services, failures, - ); - let requirements = run.requirements(prompt, &packs.unresolved); - (run, requirements) + &packs.unresolved, + ) } } diff --git a/crates/harness-internal/runner/src/lib.rs b/crates/harness-internal/runner/src/lib.rs index eebc7a1c6..b1cee226e 100644 --- a/crates/harness-internal/runner/src/lib.rs +++ b/crates/harness-internal/runner/src/lib.rs @@ -1,13 +1,15 @@ //! harness-runner - the per-run Harness and the effect loop it drives: a -//! [`Harness`] resolves the launch model through the Host's broker, -//! prepares an Engine `Run` from the prompt's source (drawing the inputs -//! the Engine refuses to draw itself, putting the declared input file in -//! place, taking the run's snapshot of the [`HostContext`]'s Packs, -//! beginning the run at its recorder), steps it, performs each chat, -//! render, tool-call, and timer effect through its performer and answers -//! each Vfs effect inline, feeds the answers back, hands every event, -//! effect, and answer to the run's recorder, and reads the declared output -//! file, all inside the one future [`Harness::run`] returns. +//! [`Harness`] begins the run at its recorder with the inputs the Engine +//! refuses to draw itself, resolves the launch model through the Host's +//! broker, prepares an Engine `Run` from the prompt's source (putting the +//! declared input file in place, taking the run's snapshot of the +//! [`HostContext`]'s Packs), steps it, performs each chat, render, +//! tool-call, and timer effect through its performer and answers each Vfs +//! effect inline, feeds the answers back, hands every event, effect, and +//! answer to the run's recorder, ends the run there with its one outcome, +//! and reads the declared output file, all inside the one future +//! [`Harness::run`] returns. A recorder that refuses a write is the only +//! way a run ends without a recorded outcome. //! //! ## Invariants //! @@ -23,8 +25,8 @@ //! each answer before the run resumes with it. Every effect record has //! exactly one answer record; a dropped effect's answer is `Dropped`. //! - The loop never reads an event to decide anything; control comes -//! from the run's own word (`Step`, `Run::decided`), the cancel flag, -//! and the Host's stop. +//! from the run's own word (`Step` and the effects it withdraws), the +//! cancel flag, and the Host's stop. //! - A [`HostContext`] builds every Pack its `PackRegistry` holds, and //! only those. The registry holds the built-in `runvil/subagents` from //! the start and refuses a duplicate FQN and the built-in's Pack id to @@ -47,6 +49,6 @@ pub use display_chain::display_chain; pub use harness::{Harness, HarnessError, RunControl, RunReport, RunRequest}; pub use host::HostContext; pub use recorder::{ - MemoryRecorder, Record, RecordKind, RecorderError, RecorderFuture, RunId, RunMeta, RunOutcome, - RunRecorder, + MemoryRecorder, Record, RecordKind, RecorderError, RecorderFuture, RunErrorKind, RunId, + RunMeta, RunOutcome, RunRecorder, }; diff --git a/crates/harness-internal/runner/src/prepare.rs b/crates/harness-internal/runner/src/prepare.rs index e23f112ca..fdb7cb994 100644 --- a/crates/harness-internal/runner/src/prepare.rs +++ b/crates/harness-internal/runner/src/prepare.rs @@ -1,31 +1,31 @@ //! Run preparation: everything between a prompt's source text and a `Run` //! the effect loop can drive. //! -//! Here the Harness draws the inputs the Engine refuses to draw itself: -//! the run's seed from the OS CSPRNG and its `started_at` from the wall -//! clock, both handed to the recorder when the run begins, before anything -//! else, so the record holds them verbatim. Then -//! the ceremony `RunContext::prepare` expects of the Harness: -//! parse; put the prompt's declared `input:` file in place in the store -//! (`files::stage_input`); hand the run's whole filesystem, real -//! directories and the declared store, to the context as given; wait for -//! every built Pack to be ready, or for the run's cancel; take the -//! run's snapshot of the Host's Packs with the run's own services, -//! which yields the catalog, the roster, the preludes, and the tool -//! performer; set the first three on the context and prepare it; merge -//! the snapshot's report into prepare's and refuse an unsatisfiable -//! prompt with the Engine's model-readable notice; and build the `Run` -//! beside its performers. +//! [`begin`] draws the inputs the Engine refuses to draw itself: the +//! run's seed from the OS CSPRNG and its `started_at` from the wall +//! clock, both handed to the recorder as the run begins, so the record +//! holds them verbatim. `Harness::run_to_end` begins the run before +//! anything else, so every run has a record. Then [`prepare`] takes the +//! steps that build the run: parse; put the prompt's declared `input:` +//! file in place in the store (`files::stage_input`); hand the run's +//! whole filesystem, real directories and the declared store, to the +//! config as given; wait for every built Pack to be ready, or for the +//! run's cancel; take the run's snapshot of the Host's Packs with the +//! run's own services, which yields the catalog, the roster, the +//! preludes, and the tool performer; set the first three and the Host's +//! model on the config; and build the `Run` beside its performers. //! -//! A refusal (or a prompt that fails to parse, or an input file that -//! cannot be put in place) is a run that ended before -//! it began: the recorder ends it as failed with the refusal as the -//! message, so the record answers "why did this run fail" for a run the -//! loop never saw. A refusal after the run's cancel fired, as when a -//! cancel cuts the wait short while a needed Pack is still starting, -//! ends the run as cancelled instead. `Harness::run_to_end` reports that -//! ended run, and its caller reads the events recorded so far from the -//! recorder. +//! A prompt that fails to parse, or an input file that cannot be put in +//! place, is a run that ended before it began: the recorder ends it as +//! failed with the cause as the message, so the record answers "why did +//! this run fail" for a run the loop never saw. `Harness::run_to_end` +//! reports that ended run, and its caller reads the events recorded so +//! far from the recorder. An environment that cannot satisfy the prompt +//! fails no preparation: the run is built, and its first step ends it +//! with the Engine's model-readable notice, or as cancelled when the +//! run's cancel fired first, as when a cancel cuts the wait short while a +//! needed Pack is still starting. The effect loop records that end like +//! any other, and the record holds no run event after the parse events. //! //! The input staging is store work and runs inline, like every other VFS //! operation of the run. @@ -36,20 +36,21 @@ use std::time::{SystemTime, UNIX_EPOCH}; use runvil::cancel::CancelHandle; use runvil::event::Event; -use runvil::model::ModelInfo; +use runvil::model::{ModelInfo, ModelRole}; use runvil::prompt::FileDecl; use runvil::timestamp::Timestamp; use runvil::vfs::VfsRef; -use runvil::{ParseError, Prompt, Requirements, Run, RunContext, RunError, RunSeed}; +use runvil::{ParseError, Prompt, Run, RunConfig, RunSeed}; use runvil_plugin::HostServices; use sha2::{Digest as _, Sha256}; use crate::display_chain::display_chain; -use crate::effect_loop::failed_outcome; use crate::files::{InputFileError, stage_input}; -use crate::host::{HostContext, HostRunContext}; +use crate::host::HostContext; use crate::performers::{InferenceBroker, Performers, Timer}; -use crate::recorder::{Record, RecordKind, RecorderError, RunId, RunMeta, RunOutcome, RunRecorder}; +use crate::recorder::{ + Record, RecordKind, RecorderError, RunErrorKind, RunId, RunMeta, RunOutcome, RunRecorder, +}; /// What the caller owns and preparation borrows: the Host's built /// Packs and the run's own services, the real directories, the run's @@ -63,25 +64,27 @@ pub struct Services { /// of the run is lent them. pub services: HostServices, /// The run's whole filesystem: the real directories and the declared store, - /// passed straight to the context's VFS. + /// passed straight to the config's VFS. pub vfs: VfsRef, /// The text staged at the prompt's declared `input:` path before the /// run, when the launch supplied one. pub input_text: Option, - /// The run's cancel flag: handed to the context and polled by the + /// The run's cancel flag: handed to the config and polled by the /// Engine. pub cancel: CancelHandle, - /// The recorder the run begins at and the loop will write to. + /// The recorder the run began at, which preparation and the loop + /// write to. pub recorder: Arc, /// Performs the run's `Chat` effects. pub broker: Arc, /// Performs the run's `Timer` effects. pub timer: Arc, - /// The run's name: the run metadata's `name` and every event's + /// The run's name, the one it began under: every event's /// `execution`. pub name: String, - /// The Host's current model, when one is selected; prepare binds - /// every declared role to it and checks each role's requirements. + /// The Host's default model, when one is selected: the run fills all + /// five roles with it, and its first step checks each declared role's + /// requirements against it. pub model: Option, /// The Host snapshot the `ui()` global serves, when the run has /// one. @@ -100,11 +103,22 @@ impl fmt::Debug for Services { } } +/// A run begun at its recorder, with the seed and start its record holds. +#[derive(Debug, Clone, Copy)] +pub struct Begun { + /// The run's id at the recorder, begun and still open. + pub run_id: RunId, + /// The seed the Harness drew for the run. + pub seed: RunSeed, + /// When the run started, as the Harness read the wall clock. + pub started_at: Timestamp, +} + /// A run ready for the effect loop, whose seed, start, and parse events are /// already at the recorder. #[derive(Debug)] pub struct Prepared { - /// The run, built over the prepared context. + /// The run, built over the prepared config. pub run: Run, /// The run's id at the recorder, begun and still open, for /// `Harness::run_to_end` to drive; the run's events, its parse events @@ -123,8 +137,8 @@ pub struct Prepared { #[derive(Debug, thiserror::Error)] #[non_exhaustive] pub enum PrepareError { - /// The prompt does not parse. The run is ended as failed; the parse - /// failure is the source. + /// The prompt does not parse. The run is ended as failed with kind + /// `Parse`; the parse failure is the source. #[error("the prompt does not parse")] Parse { /// The run, ended with this failure. @@ -133,24 +147,11 @@ pub enum PrepareError { #[source] source: ParseError, }, - /// The environment cannot satisfy the prompt: a Pack the prompt - /// declares is missing, unavailable, or lacks a service it needs, or - /// the current model falls short of a role's requirements. The Engine's - /// model-readable notice, one line per gap, is the source; the run is - /// ended as failed with that notice, or as cancelled when the run's - /// cancel had fired. - #[error("the environment cannot satisfy the prompt")] - Refused { - /// The run, ended with this refusal. - run_id: RunId, - /// The refusal, of kind `RequirementsUnmet`. - #[source] - error: RunError, - }, /// The prompt's declared input file could not be put in place: the /// launch supplied text the prompt declares no file for, the prompt /// declares a file that is neither supplied nor in the store, or the - /// store refused. The run is ended as failed with kind `Input`. + /// store refused. The run is ended as failed with kind + /// `RequirementsUnmet`, or `Vfs` when the store refused. #[error("the prompt's declared input file cannot be put in place")] Input { /// The run, ended with this refusal. @@ -160,12 +161,11 @@ pub enum PrepareError { source: InputFileError, }, /// The recorder refused a write; the run cannot be recorded, so it is - /// not prepared. `run` is the run the recorder issued, `None` when it - /// refused to begin one. + /// not prepared. #[error("the run could not be recorded")] Recorder { - /// The run the recorder issued before it refused. - run: Option, + /// The run the recorder refused a write for. + run: RunId, /// The recorder's refusal. #[source] source: RecorderError, @@ -173,67 +173,72 @@ pub enum PrepareError { } impl PrepareError { - /// The run this failure ended at the recorder, beside the failed - /// outcome it was ended with, or, for a recorder refusal, which ends - /// nothing, the run the recorder issued and its refusal. A refusal - /// that [`prepare_noting_cancel`] says it ended `Cancelled` was not - /// ended as failed, and its caller reports it without asking here. - pub(crate) fn ended(self) -> Result<(RunId, RunOutcome), (Option, RecorderError)> { + /// The failed outcome this failure ended the run with, or, for a + /// recorder refusal, which ends nothing, the refusal. + pub(crate) fn ended(self) -> Result { match self { - PrepareError::Parse { run_id, source } => Ok((run_id, failed("Parse", &source))), - PrepareError::Refused { run_id, error } => Ok((run_id, failed_outcome(&error))), - PrepareError::Input { run_id, source } => Ok((run_id, failed("Input", &source))), - PrepareError::Recorder { run, source } => Err((run, source)), + PrepareError::Parse { source, .. } => Ok(failed(RunErrorKind::Parse, &source)), + PrepareError::Input { source, .. } => Ok(failed(source.kind(), &source)), + PrepareError::Recorder { source, .. } => Err(source), } } } -/// Prepares the prompt `source` for one run with `args`: draws the run's -/// seed and start and begins the run at its recorder, parses the prompt, -/// puts its declared input file in place, waits for every built Pack -/// to be ready or for the run's cancel, takes the run's snapshot of the -/// Host's Packs, sets the snapshot's catalog, roster, and declared -/// preludes on the context and prepares it, refuses an unsatisfiable -/// prompt, and builds the `Run` and its performers. +/// Begins one run named `name` of the prompt `source` at `recorder`: draws +/// the run's seed and start and hands both to the recorder with the +/// prompt's hash. +/// +/// # Errors +/// Returns the recorder's refusal to begin the run. +pub async fn begin( + recorder: &dyn RunRecorder, + name: &str, + source: &str, +) -> Result { + let seed = RunSeed::from_bytes(rand::random()); + let started_at = now_timestamp(); + let run_id = recorder + .begin_run(RunMeta { + name: name.to_owned(), + prompt_hash: prompt_hash(source), + seed, + flags: 0, + started_at: started_at.unix_millis(), + }) + .await?; + Ok(Begun { + run_id, + seed, + started_at, + }) +} + +/// Prepares the prompt `source` for the run `begun` with `args`: parses +/// the prompt, puts its declared input file in place, waits for every +/// built Pack to be ready or for the run's cancel, takes the run's +/// snapshot of the Host's Packs, sets the snapshot's catalog, roster, and +/// declared preludes and the Host's model on the config, and builds the +/// `Run` and its performers. /// /// A Pack whose `ready` fails is unavailable to this run, with its /// error as the reason. A cancel ends the wait, and each Pack is -/// snapshotted as it stands. +/// snapshotted as it stands. A prompt the environment cannot satisfy is +/// prepared all the same: the run's first step ends it with the Engine's +/// notice, or as cancelled when the run's cancel fired first. /// /// # Errors -/// Returns [`PrepareError::Parse`] when the source does not parse, +/// Returns [`PrepareError::Parse`] when the source does not parse and /// [`PrepareError::Input`] when its declared input file cannot be put in -/// place, and [`PrepareError::Refused`] when the environment cannot -/// satisfy it (in these three cases the run is ended as failed, except -/// that a refusal after the run's cancel fired ends it as cancelled, and +/// place (in both cases the run is ended as failed, and /// `Harness::run_to_end` reports it while the events recorded so far stay /// at the recorder), and [`PrepareError::Recorder`] when the recorder /// refuses a write. pub async fn prepare( source: &str, args: &str, + begun: Begun, services: Services, ) -> Result { - prepare_noting_cancel(source, args, services) - .await - .map_err(|(error, _)| error) -} - -/// [`prepare`], whose failure comes beside whether it ended the run -/// `Cancelled`, which only a refusal after the run's cancel fired does. -/// That one reading of the cancel decides the record, and the caller -/// reports from it rather than reading the cancel again, so a cancel -/// landing while the refusal is recorded cannot split the report from -/// the record. -#[expect( - clippy::result_large_err, - reason = "the error is returned once per launch and carries the failure's source beside the one cancel reading" -)] -pub(crate) async fn prepare_noting_cancel( - source: &str, - args: &str, - services: Services, -) -> Result { let Services { host, services, @@ -247,29 +252,14 @@ pub(crate) async fn prepare_noting_cancel( model, ui, } = services; - - // The inputs the Engine leaves to the Harness, recorded before the - // run exists so the record has them however the run ends. - let seed = RunSeed::from_bytes(rand::random()); - let started_at = now_timestamp(); - let run_id = recorder - .begin_run(RunMeta { - name: name.clone(), - prompt_hash: prompt_hash(source), - seed, - flags: 0, - started_at: started_at.unix_millis(), - }) - .await - .map_err(|source| (PrepareError::Recorder { run: None, source }, false))?; - let recorded = |source| { - ( - PrepareError::Recorder { - run: Some(run_id), - source, - }, - false, - ) + let Begun { + run_id, + seed, + started_at, + } = begun; + let recorded = |source| PrepareError::Recorder { + run: run_id, + source, }; // Parse-time events are the run's first records, whether or not the @@ -283,10 +273,10 @@ pub(crate) async fn prepare_noting_cancel( Ok(prompt) => prompt, Err(source) => { recorder - .end_run(run_id, failed("Parse", &source)) + .end_run(run_id, failed(RunErrorKind::Parse, &source)) .await .map_err(recorded)?; - return Err((PrepareError::Parse { run_id, source }, false)); + return Err(PrepareError::Parse { run_id, source }); } }; @@ -295,10 +285,10 @@ pub(crate) async fn prepare_noting_cancel( let declared = prompt.frontmatter().input().map(FileDecl::path); if let Err(source) = stage_input(&vfs, declared, input_text) { recorder - .end_run(run_id, failed("Input", &source)) + .end_run(run_id, failed(source.kind(), &source)) .await .map_err(recorded)?; - return Err((PrepareError::Input { run_id, source }, false)); + return Err(PrepareError::Input { run_id, source }); } let output_path = prompt .frontmatter() @@ -309,37 +299,35 @@ pub(crate) async fn prepare_noting_cancel( // root task continues the sequence past them, so `(task_id, task_seq)` // is unique across every record of the run. let provenance_start = u32::try_from(parse_events.len()).unwrap_or(u32::MAX); - let mut ctx = RunContext::new(name, seed, started_at) + let mut ctx = RunConfig::new(name, seed, started_at) .with_cancel(cancel.clone()) .with_provenance_start(provenance_start); if let Some(ui) = ui { ctx = ctx.with_ui(ui); } - // The pick-wait-snapshot-prepare-refuse ceremony: the run's whole - // filesystem, the real directories and the declared store, goes to - // the context as given; the run's Packs are picked from the prompt's - // `uses:` entries; the snapshot is taken once each of them is ready - // or the run is cancelled, its catalog is what the run offers, its - // roster gives each Pack's object in the run's Lua, its preludes go - // to every section VM, and the snapshot itself stays here as the tool - // performer, lending each call the run's services. A refusal after a - // cancel ends the run cancelled, as `Harness::run_to_end` reports it. + // The pick-wait-snapshot ceremony: the run's whole filesystem, the + // real directories and the declared store, goes to the config as + // given; the run's Packs are picked from the prompt's `uses:` entries; + // the snapshot is taken once each of them is ready or the run is + // cancelled, its catalog is what the run offers, its roster gives each + // Pack's object in the run's Lua and is what the run's first step + // checks the declared Packs against, its preludes go to every section + // VM, and the snapshot itself stays here as the tool performer, + // lending each call the run's services. let packs = host.pick(&prompt); let failures = host.wait_ready(&packs, &cancel).await; - let (snapshot, unmet) = host.begin_run(&packs, services, failures, &prompt); - let (ctx, mut requirements) = - prepare_from_snapshot(ctx.with_vfs(vfs), &snapshot, &prompt, model); - requirements.merge(unmet); - if let Some(error) = requirements.refusal() { - let cancelled = cancel.is_cancelled(); - let outcome = if cancelled { - RunOutcome::Cancelled - } else { - failed_outcome(&error) - }; - recorder.end_run(run_id, outcome).await.map_err(recorded)?; - return Err((PrepareError::Refused { run_id, error }, cancelled)); - } + let snapshot = host.begin_run(&packs, services, failures); + let ctx = ctx + .with_vfs(vfs) + .with_tools(snapshot.catalog()) + .with_packs(snapshot.roster()) + .with_preludes(snapshot.preludes(&prompt)) + .with_models( + model + .into_iter() + .map(|model| (ModelRole::Default, model)) + .collect(), + ); let run = Run::new(Arc::new(prompt), args, ctx); let performers = Performers { @@ -355,27 +343,11 @@ pub(crate) async fn prepare_noting_cancel( }) } -/// Sets `snapshot`'s catalog, roster with each usable Pack's vendor, and -/// declared preludes on `ctx`, then prepares it for `prompt` under the -/// current `model`. -fn prepare_from_snapshot( - ctx: RunContext, - snapshot: &HostRunContext, - prompt: &Prompt, - model: Option, -) -> (RunContext, Requirements) { - let (usable, unavailable) = snapshot.roster(); - ctx.with_tools(snapshot.catalog()) - .with_packs(usable, unavailable) - .with_preludes(snapshot.preludes(prompt)) - .prepare(prompt, model) -} - -/// The failed outcome of a run that ended in preparation: `kind` names the -/// stage and the message is the cause chain. -fn failed(kind: &str, source: &dyn std::error::Error) -> RunOutcome { +/// The failed outcome of a run that ended in preparation: `kind` is the +/// Engine's kind for the failure and the message is the cause chain. +fn failed(kind: RunErrorKind, source: &dyn std::error::Error) -> RunOutcome { RunOutcome::Failed { - kind: kind.to_owned(), + kind, message: display_chain(source), } } diff --git a/crates/harness-internal/runner/src/recorder.rs b/crates/harness-internal/runner/src/recorder.rs index bba89fd17..1a4145ba9 100644 --- a/crates/harness-internal/runner/src/recorder.rs +++ b/crates/harness-internal/runner/src/recorder.rs @@ -13,6 +13,14 @@ //! serializes its own state. A recorder that fails stops its run: the //! Harness never skips a record. A Host that prefers to keep running logs //! the failure inside its recorder and returns `Ok`. +//! +//! A run's end is the Engine's [`RunOutcome`], as the run's last step +//! returned it. A run that ends before the Engine runs it ends with an +//! outcome of the same type: cancelled, when a cancel comes before or +//! during the model lookup, or failed with the Engine's kind for the +//! failure, when the model lookup fails, the prompt does not parse, or its +//! input cannot be put in place. A run the effect loop loses track of +//! ends as failed with the kind `Internal`. use std::error::Error; use std::fmt; @@ -20,6 +28,7 @@ use std::future::Future; use std::pin::Pin; use runvil::RunSeed; +pub use runvil::{RunErrorKind, RunOutcome}; #[path = "recorder-memory.rs"] mod memory; @@ -124,7 +133,7 @@ pub struct RunMeta { /// The seed the Harness drew and passed to the Engine. A replay passes /// it back, and the run's id derives from it. pub seed: RunSeed, - /// The Engine's behavior flags, as a bitset. + /// Reserved. No flag is defined, and the Harness records 0. pub flags: u32, /// When the run started, in UTC milliseconds since the Unix epoch. /// The Engine receives the same value as `started_at`, so the record @@ -135,10 +144,11 @@ pub struct RunMeta { /// Which side of the effect loop a record came from. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum RecordKind { - /// An effect the Engine issued. The record's payload is an - /// `EffectRecord`. + /// An effect the Engine issued. The record's payload is the JSON + /// `Effect::record` returns. Effect, - /// The answer to an effect. The record's payload is an `EffectAnswer`. + /// The answer to an effect. The record's payload is the JSON + /// `EffectAnswer::record` returns. Answer, /// An event the Engine emitted. The record's payload is an `Event`. Event, @@ -185,38 +195,6 @@ pub struct Record { pub kind: RecordKind, /// The in-flight effect handle, for effects and their answers. pub effect_id: Option, - /// The serialized `EffectRecord`, `EffectAnswer`, or `Event`. + /// The effect's or answer's record, or the serialized `Event`. pub payload: serde_json::Value, } - -/// How a run ended. Mirrors the Engine's `RunResult`. -#[derive(Debug, Clone, PartialEq, Eq)] -pub enum RunOutcome { - /// The run completed with its final text. - Completed { - /// The run's final text. - final_text: String, - }, - /// The run failed. - Failed { - /// The failure's classification. - kind: String, - /// The failure's message. - message: String, - }, - /// The Host cancelled the run. - Cancelled, -} - -impl RunOutcome { - /// The text that stores this outcome: `completed`, `failed`, or - /// `cancelled`. - #[must_use] - pub const fn as_str(&self) -> &'static str { - match self { - Self::Completed { .. } => "completed", - Self::Failed { .. } => "failed", - Self::Cancelled => "cancelled", - } - } -} diff --git a/crates/harness-internal/runner/tests/it/effect_loop-broker.rs b/crates/harness-internal/runner/tests/it/effect_loop-broker.rs index a0903eb93..32285b74d 100644 --- a/crates/harness-internal/runner/tests/it/effect_loop-broker.rs +++ b/crates/harness-internal/runner/tests/it/effect_loop-broker.rs @@ -3,6 +3,7 @@ //! for a nested `models.infer` and `Chat` for a section's own round, and //! the run resumes with the broker's completion as each round's answer. +use std::collections::BTreeMap; use std::num::NonZeroU32; use std::sync::{Arc, Mutex}; @@ -15,10 +16,10 @@ use runvil::event::ReplyOrigin; use runvil::ids::RoundId; use runvil::model::{ Completion, CompletionError, CompletionOptions, CompletionResult, Message, ModelBinding, - ModelCatalog, ModelId, ModelInfo, ThinkingMode, ToolSchema, + ModelCatalog, ModelId, ModelInfo, ModelRole, ThinkingMode, ToolSchema, }; use runvil::timestamp::Timestamp; -use runvil::{Prompt, Run, RunContext}; +use runvil::{Prompt, Run, RunConfig}; use serde_json::{Value, json}; use super::begun_log; @@ -27,8 +28,8 @@ use crate::support::{EXECUTION, TEST_SEED, unused}; /// A program that infers once and then chats once: the nested round /// first, then the section's own round over its message list. const INFERS_THEN_CHATS: &str = "---\nname: runner-broker\ndescription: infers then chats\n\ - runvil: 0\nmodels:\n writer: {}\n---\n\n\ - # Broker\n\n```lua\nmodels.default('writer')\n```\n\n\ + runvil: 0\n---\n\n\ + # Broker\n\n\ ## Only\n\n```lua\n\ local inferred = models.infer('infer round')\n\ messages:user('chat round')\n\ @@ -36,7 +37,7 @@ const INFERS_THEN_CHATS: &str = "---\nname: runner-broker\ndescription: infers t return inferred .. '|' .. messages[#messages].content\n\ ```\n"; -/// A run over [`INFERS_THEN_CHATS`] with every declared role bound to one +/// A run over [`INFERS_THEN_CHATS`] with the `default` role filled by one /// current model. fn infers_then_chats() -> Run { let (prompt, _parse_events) = Prompt::parse(INFERS_THEN_CHATS, EXECUTION); @@ -47,12 +48,8 @@ fn infers_then_chats() -> Run { NonZeroU32::new(131_072).expect("131072 is non-zero"), ThinkingMode::Never, ); - let ctx = RunContext::new(EXECUTION, TEST_SEED, Timestamp::UNIX_EPOCH); - let (ctx, requirements) = ctx.prepare(&prompt, Some(model)); - assert!( - requirements.is_satisfied(), - "the bound model satisfies the fixture's one role: {requirements:?}" - ); + let ctx = RunConfig::new(EXECUTION, TEST_SEED, Timestamp::UNIX_EPOCH) + .with_models(BTreeMap::from([(ModelRole::Default, model)])); Run::new(prompt, "", ctx) } diff --git a/crates/harness-internal/runner/tests/it/effect_loop-cap.rs b/crates/harness-internal/runner/tests/it/effect_loop-cap.rs index 60c8d31f9..cdbb88770 100644 --- a/crates/harness-internal/runner/tests/it/effect_loop-cap.rs +++ b/crates/harness-internal/runner/tests/it/effect_loop-cap.rs @@ -3,6 +3,7 @@ //! waiting call that a withdrawal, a task cancel, or a run cancel reaches //! is answered `Dropped` without ever starting. +use std::collections::BTreeMap; use std::future::Future; use std::num::{NonZeroU32, NonZeroUsize}; use std::sync::Arc; @@ -15,12 +16,13 @@ use harness_runner::recorder::{MemoryRecorder, RecordKind, RunId, RunOutcome}; use runvil::cancel::CancelHandle; use runvil::effect::ToolCallOrigin; use runvil::model::{ - Completion, CompletionError, CompletionResult, ModelId, ModelInfo, ThinkingMode, ToolCall, + Completion, CompletionError, CompletionResult, ModelId, ModelInfo, ModelRole, ThinkingMode, + ToolCall, }; use runvil::timestamp::Timestamp; use runvil::tools::{ToolCatalog, ToolError, ToolId, ToolInfo, ToolOutput}; use runvil::vfs::Access; -use runvil::{Prompt, Run, RunContext}; +use runvil::{Prompt, Run, RunConfig}; use serde_json::{Value, json}; use tokio::sync::{Notify, Semaphore}; @@ -37,11 +39,10 @@ const TOOL: &str = "tests/cap/park"; /// [`TOOL`]'s wire name, as the model calls it. const WIRE: &str = "tests_cap_park"; -/// The frontmatter and setup every run here shares: one model role bound -/// as the default. +/// The frontmatter and title every run here shares. const PREAMBLE: &str = "---\nname: runner-cap\ndescription: caps tool calls\nrunvil: 0\n\ - models:\n writer: {}\n---\n\n\ - # Cap\n\n```lua\nmodels.default('writer')\n```\n\n"; + ---\n\n\ + # Cap\n\n"; /// Main spawns Child and waits on a `wait` round while Child's loop asks /// its round, then cancels Child and lingers on a second round. @@ -68,9 +69,9 @@ fn looping(section: &str, user: &str, result: &str) -> String { ) } -/// A run over [`PREAMBLE`] and `sections`, with the one role bound to one -/// current model and [`TOOL`] in its catalog. -fn run_of(sections: &str) -> Run { +/// A run over [`PREAMBLE`] and `sections`, with the `default` role filled by +/// one model and [`TOOL`] in its catalog, under the cancel flag `cancel`. +fn run_of(sections: &str, cancel: &CancelHandle) -> Run { let source = format!("{PREAMBLE}{sections}"); let (prompt, _parse_events) = Prompt::parse(&source, EXECUTION); let prompt = Arc::new(prompt.expect("the cap fixture parses")); @@ -86,12 +87,10 @@ fn run_of(sections: &str) -> Run { json!({ "type": "object", "properties": { "name": { "type": "string" } } }), ); let catalog = ToolCatalog::new([tool]).expect("the one-tool catalog is valid"); - let ctx = RunContext::new(EXECUTION, TEST_SEED, Timestamp::UNIX_EPOCH).with_tools(catalog); - let (ctx, requirements) = ctx.prepare(&prompt, Some(model)); - assert!( - requirements.is_satisfied(), - "the bound model and the catalog satisfy the fixture: {requirements:?}" - ); + let ctx = RunConfig::new(EXECUTION, TEST_SEED, Timestamp::UNIX_EPOCH) + .with_tools(catalog) + .with_models(BTreeMap::from([(ModelRole::Default, model)])) + .with_cancel(cancel.clone()); Run::new(prompt, "", ctx) } @@ -237,11 +236,12 @@ impl Rig { } } - /// Drives `run` with `broker` and the rig's tool beside `steer`, and - /// returns the run's outcome. Fails when the two outlast [`PATIENCE`]. + /// Drives a run over `sections` under the rig's cancel flag, with + /// `broker` and the rig's tool, beside `steer`, and returns the run's + /// outcome. Fails when the two outlast [`PATIENCE`]. async fn drive( &self, - run: Run, + sections: &str, broker: ScriptedBroker, steer: impl Future, ) -> RunOutcome { @@ -249,7 +249,7 @@ impl Rig { performers.broker = Arc::new(broker); performers.tool = self.tool.clone(); let drive = drive_run( - run, + run_of(sections, &self.cancel), performers, self.recorder.clone(), self.run_id, @@ -309,7 +309,7 @@ async fn the_cap_holds_extra_calls_until_one_lands() { }); let outcome = rig - .drive(run_of(&looping("Only", "go", "done")), broker, async { + .drive(&looping("Only", "go", "done"), broker, async { until("two calls start", || rig.tool.started().len() >= 2).await; settle().await; assert_eq!( @@ -350,7 +350,7 @@ async fn waiting_calls_start_oldest_first() { ); let outcome = rig - .drive(run_of(§ions), broker, async { + .drive(§ions, broker, async { until("B's call is issued", || rig.recorded("b1")).await; settle().await; assert_eq!(rig.tool.started(), ["a1"], "the cap holds the rest"); @@ -387,7 +387,7 @@ async fn a_call_withdrawn_in_the_step_that_issued_it_never_starts() { }); let sections = format!("{MAIN_CANCELS}{}", looping("Child", "go", "child done")); - let outcome = rig.drive(run_of(§ions), broker, async {}).await; + let outcome = rig.drive(§ions, broker, async {}).await; assert_eq!(outcome, completed("cancelled")); assert!( @@ -418,7 +418,7 @@ async fn cancelling_a_task_drops_its_waiting_and_running_calls() { }); let sections = format!("{MAIN_CANCELS}{}", looping("Child", "go", "child done")); - let outcome = rig.drive(run_of(§ions), broker, async {}).await; + let outcome = rig.drive(§ions, broker, async {}).await; assert_eq!(outcome, completed("cancelled")); assert_eq!(rig.tool.started(), ["c1"], "the waiting call never started"); @@ -442,7 +442,7 @@ async fn a_cancel_drops_waiting_calls_without_starting_them() { }); let outcome = rig - .drive(run_of(&looping("Only", "go", "done")), broker, async { + .drive(&looping("Only", "go", "done"), broker, async { until("one call starts", || !rig.tool.started().is_empty()).await; rig.cancel.cancel(); }) diff --git a/crates/harness-internal/runner/tests/it/effect_loop-recorder.rs b/crates/harness-internal/runner/tests/it/effect_loop-recorder.rs index 74c25a284..286e80c35 100644 --- a/crates/harness-internal/runner/tests/it/effect_loop-recorder.rs +++ b/crates/harness-internal/runner/tests/it/effect_loop-recorder.rs @@ -1,5 +1,5 @@ //! A recorder that refuses a write ends the drive: whichever call it -//! refuses, the loop returns `DriveError::Recorder` and sends the recorder +//! refuses, the loop returns the recorder's error and sends the recorder //! nothing more, and a refusal mid-run aborts the performers still out. use std::sync::Arc; @@ -7,14 +7,15 @@ use std::sync::atomic::AtomicBool; use std::time::Duration; use harness_runner::display_chain; -use harness_runner::effect_loop::{DriveError, drive_run}; +use harness_runner::effect_loop::drive_run; use harness_runner::recorder::{RecordKind, RunRecorder}; use runvil::cancel::CancelHandle; use runvil::vfs::VfsRef; use super::{answers, await_raised, cancel_after, meta}; use crate::support::{ - FailingRecorder, PendingTimer, TIMED_MAIN, TextTool, WAITS, run_over, run_with_child, unused, + FailingRecorder, PendingTimer, TIMED_MAIN, TextTool, WAITS, run_over, run_with_child, + run_with_child_and_cancel, unused, }; /// A run of one store write and one read, over a fresh memory store: its @@ -56,10 +57,6 @@ async fn a_recorder_that_refuses_its_nth_call_ends_the_drive_with_a_recorder_err .await .expect_err("a refused call ends the drive"); - assert!( - matches!(error, DriveError::Recorder(_)), - "call {call} of {total}: {error:?}" - ); assert!( display_chain(&error).contains(&format!("refuses call {call}")), "call {call} of {total}: {}", @@ -98,7 +95,7 @@ async fn a_recorder_that_refuses_an_answer_aborts_the_performers_still_out() { let cancel = CancelHandle::new(); cancel_after(&cancel, Duration::from_millis(50)); drive_run( - run_with_child(TIMED_MAIN, WAITS), + run_with_child_and_cancel(TIMED_MAIN, WAITS, &cancel), performers, dry.clone(), dry_run, @@ -133,7 +130,11 @@ async fn a_recorder_that_refuses_an_answer_aborts_the_performers_still_out() { .await .expect_err("the refused answer ends the drive"); - assert!(matches!(error, DriveError::Recorder(_)), "got {error:?}"); + assert!( + display_chain(&error).contains(&format!("refuses call {}", first_answer + 1)), + "the chain names the recorder's reason: {}", + display_chain(&error) + ); assert!( answers(&recorder.inner().records(run_id)).is_empty(), "the refused answer was not recorded" diff --git a/crates/harness-internal/runner/tests/it/effect_loop-vfs.rs b/crates/harness-internal/runner/tests/it/effect_loop-vfs.rs index b40471edc..6d5f9c183 100644 --- a/crates/harness-internal/runner/tests/it/effect_loop-vfs.rs +++ b/crates/harness-internal/runner/tests/it/effect_loop-vfs.rs @@ -1,9 +1,11 @@ //! The effect loop answers a Vfs effect inline: a run of nothing but Vfs //! operations completes without a stall, the operation runs on the loop's //! own thread, a Vfs chain keeps stepping while another effect is parked, -//! and a backend that panics drops its effect instead of unwinding the -//! run. +//! a backend that panics drops its effect instead of unwinding the run, +//! and a Vfs effect withdrawn in the step that issued it is dropped +//! without being performed. +use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex}; use std::thread::ThreadId; use std::time::Duration; @@ -15,7 +17,9 @@ use runvil::vfs::VfsRef; use serde_json::json; use super::{answers, assert_one_answer_per_effect, begun_log, kinds}; -use crate::support::{GatedTool, HookedBackend, TextTool, WAITS, run_over, run_with_child, unused}; +use crate::support::{ + GatedTool, HookedBackend, TextTool, WAITS, run_of_sections, run_over, run_with_child, unused, +}; #[tokio::test] async fn a_run_of_only_vfs_operations_completes_with_one_answer_per_effect() { @@ -215,6 +219,67 @@ async fn a_backend_that_panics_drops_its_effect_instead_of_unwinding_the_run() { assert_eq!(recorder.outcome(run_id), Some(RunOutcome::Cancelled)); } +/// Main spawns Slow, whose write stays out, and Fast, waits on Fast, +/// cancels Slow, and returns: the step that decides the run issues Slow's +/// write and withdraws it. +const WITHDRAWS_A_WRITE: [(&str, &str); 3] = [ + ( + "Main", + "local slow = tasks.spawn('## Slow')\n\ + local fast = tasks.spawn('## Fast')\n\ + local _task, _ok, result = tasks.join_any({ fast })\n\ + tasks.cancel(slow)\n\ + return result", + ), + ("Slow", "store.write('slow.md', 'never')\nreturn 'slow'"), + ("Fast", "return 'fast'"), +]; + +#[tokio::test] +async fn a_vfs_effect_withdrawn_in_the_step_that_issued_it_is_dropped_unperformed() { + let (recorder, run_id) = begun_log().await; + let written = Arc::new(AtomicBool::new(false)); + let seen = Arc::clone(&written); + let vfs = VfsRef::builder() + .store( + "/", + HookedBackend::new(move || seen.store(true, Ordering::SeqCst)), + ) + .build(); + + let outcome = drive_run( + run_of_sections(&WITHDRAWS_A_WRITE, vfs), + unused(), + recorder.clone(), + run_id, + CancelHandle::new(), + ) + .await + .unwrap(); + assert_eq!( + outcome, + RunOutcome::Completed { + final_text: "fast".to_owned() + } + ); + assert!( + !written.load(Ordering::SeqCst), + "the withdrawn write never reached the backend" + ); + + let records = recorder.records(run_id); + assert_one_answer_per_effect(&records); + let effects: Vec<&Record> = records + .iter() + .filter(|record| record.kind == RecordKind::Effect) + .collect(); + assert_eq!(effects.len(), 1, "Slow's write is the run's one effect"); + assert_eq!(effects[0].payload, write_effect("slow.md", "never")); + let answers = answers(&records); + assert_eq!(answers.len(), 1); + assert_eq!(answers[0].payload, json!("Dropped")); +} + /// A label for one record: an event's kind, or the effect or answer kind /// with the operation's name. fn label(record: &Record) -> String { diff --git a/crates/harness-internal/runner/tests/it/effect_loop.rs b/crates/harness-internal/runner/tests/it/effect_loop.rs index 001c1b2a5..931b3404c 100644 --- a/crates/harness-internal/runner/tests/it/effect_loop.rs +++ b/crates/harness-internal/runner/tests/it/effect_loop.rs @@ -2,6 +2,7 @@ //! record stream is events, then effects, then answers per step; a cancel //! drops every outstanding effect with one `Dropped` answer each; a //! performer that panics drops its effect rather than stranding the run; +//! a run left with nothing to await ends recorded as an internal failure; //! and a refused recorder write ends the drive with the recorder's error //! and tears down the performers still out. The Vfs effect the loop //! answers inline has its own module, `vfs`, the `Chat` effect the @@ -12,17 +13,18 @@ use std::sync::atomic::{AtomicBool, Ordering}; use std::time::Duration; use harness_runner::display_chain; -use harness_runner::effect_loop::{DriveError, drive_run}; +use harness_runner::effect_loop::drive_run; use harness_runner::recorder::{ - MemoryRecorder, Record, RecordKind, RunId, RunMeta, RunOutcome, RunRecorder, + MemoryRecorder, Record, RecordKind, RunErrorKind, RunId, RunMeta, RunOutcome, RunRecorder, }; +use runvil::Step; use runvil::cancel::CancelHandle; use runvil::vfs::{MemoryBackend, Origin, VfsRef}; use serde_json::json; use crate::support::{ ClosingTool, PanickingTool, PendingTimer, PendingTool, TEST_SEED, TIMED_MAIN, TextTool, WAITS, - run, run_over, run_with_child, unused, + run, run_over, run_with_child, run_with_child_and_cancel, unused, }; #[path = "effect_loop-broker.rs"] @@ -218,7 +220,7 @@ async fn a_cancel_writes_one_dropped_answer_per_outstanding_effect() { // Two effects are out when the cancel lands: the main section's // timeout timer and its child's tool call. let outcome = drive_run( - run_with_child(TIMED_MAIN, WAITS), + run_with_child_and_cancel(TIMED_MAIN, WAITS, &cancel), performers, recorder.clone(), run_id, @@ -327,10 +329,6 @@ async fn a_refused_recorder_write_returns_the_recorder_error_and_aborts_the_park ) .await .expect_err("a refused write ends the drive"); - assert!( - matches!(error, DriveError::Recorder(_)), - "the recorder's refusal is returned as is: {error:?}" - ); assert!( display_chain(&error).contains(&format!("run {run_id} has ended")), "the chain names the recorder's reason: {}", @@ -378,6 +376,49 @@ async fn an_ended_run_refuses_the_first_write_before_any_performer_starts() { ) .await .expect_err("an ended run refuses its first write"); - assert!(matches!(error, DriveError::Recorder(_)), "got {error:?}"); + assert!( + display_chain(&error).contains(&format!("run {run_id} has ended")), + "the chain names the recorder's reason: {}", + display_chain(&error) + ); assert!(recorder.records(run_id).is_empty()); } + +#[tokio::test] +async fn a_run_left_with_nothing_to_await_ends_recorded_as_an_internal_failure() { + let (recorder, run_id) = begun_log().await; + // The test steps the run once and keeps its tool call from the loop: + // the run waits on an effect the loop never saw, the fault a lost + // effect would leave. + let mut lost = run(WAITS); + let Step::Pending { effects, .. } = lost.step() else { + panic!("the run parks on its tool call"); + }; + assert_eq!(effects.len(), 1, "the one tool call the loop never sees"); + + let outcome = drive_run( + lost, + unused(), + recorder.clone(), + run_id, + CancelHandle::new(), + ) + .await + .expect("a stall is the run's outcome"); + assert_eq!( + outcome, + RunOutcome::Failed { + kind: RunErrorKind::Internal, + message: "the effect loop has nothing to await for a pending run".to_owned(), + } + ); + assert_eq!( + recorder.outcome(run_id), + Some(outcome), + "the recorder ends the stalled run with its outcome" + ); + assert!( + recorder.records(run_id).is_empty(), + "the stalled step reported nothing and issued nothing" + ); +} diff --git a/crates/harness-internal/runner/tests/it/harness-errors.rs b/crates/harness-internal/runner/tests/it/harness-errors.rs index f372654eb..38b9abf06 100644 --- a/crates/harness-internal/runner/tests/it/harness-errors.rs +++ b/crates/harness-internal/runner/tests/it/harness-errors.rs @@ -1,19 +1,20 @@ -//! What `run` returns when the run cannot reach an outcome of its own: a -//! model list the broker cannot fetch is `HarnessError::Model`, a cancel -//! before `run` or while the broker holds its list is a cancelled report -//! with no run, a -//! recorder that refuses a write is `HarnessError::Recorder` holding the -//! run id it issued, and a broker that panics answers its round -//! `Dropped`. +//! How `run` ends a run that never reaches the Engine, and what it +//! returns when the recorder refuses: a model list the broker cannot +//! fetch ends the recorded run as a `Completion` failure, a list without +//! the selection as a `RequirementsUnmet` failure, and a cancel before +//! `run` or while the broker holds its list as cancelled, each before the +//! prompt is parsed; a recorder that refuses a write is +//! `HarnessError::Recorder` holding the run id it issued; and a broker +//! that panics answers its round `Dropped`. use std::sync::Arc; use std::sync::atomic::Ordering; use std::task::Poll; use harness_runner::display_chain; -use harness_runner::environment::CurrentModelError; -use harness_runner::recorder::MemoryRecorder; -use harness_runner::{Harness, HarnessError, HostContext}; +use harness_runner::environment::{CurrentModelError, HostSnapshot}; +use harness_runner::recorder::{MemoryRecorder, RunErrorKind}; +use harness_runner::{Harness, HarnessError, HostContext, RunReport, RunRequest}; use runvil::model::{Completion, CompletionError, CompletionErrorKind}; use runvil_plugin::{HostServices, PackRegistry}; @@ -34,8 +35,8 @@ const STORES: &str = "---\nname: stores\ndescription: d\nrunvil: 0\n---\n\n\ /// A prompt whose one round sits under a `pcall` that reports how it /// ended. const CATCHES_A_ROUND: &str = "---\nname: catches\ndescription: d\nrunvil: 0\n\ - models:\n writer: {}\n---\n\n\ - # Title\n\n```lua\nmodels.default('writer')\n```\n\n## Only\n\n```lua\n\ + ---\n\n\ + # Title\n\n## Only\n\n```lua\n\ local ok, err = pcall(models.infer, 'asked')\n\ return tostring(ok) .. ':' .. err.kind\n```\n"; @@ -51,32 +52,74 @@ fn harness_over(recorder: &Arc, broker: ScriptedBroker) -> Harn ) } +/// Asserts that `report` is of a run that ended before its prompt was +/// parsed: the recorder began it, ended it with the reported outcome, and +/// holds no record between, and the report has no output. +#[track_caller] +fn assert_ended_before_the_parse(recorder: &FailingRecorder, report: &RunReport) { + assert_eq!(recorder.begun(), [report.run_id], "the run began"); + assert_eq!( + recorder.inner().outcome(report.run_id), + Some(report.outcome.clone()), + "the recorder ends the run with the reported outcome" + ); + assert!( + recorder.inner().records(report.run_id).is_empty(), + "the run ended before its prompt was parsed" + ); + assert_eq!(recorder.calls(), 2, "the run's begin and its end"); + assert_eq!(report.output, Err(OutputError::NotCompleted)); +} + #[tokio::test] -async fn a_model_list_the_broker_cannot_fetch_returns_the_model_error_and_begins_no_run() { +async fn a_model_list_the_broker_cannot_fetch_ends_the_recorded_run_as_a_completion_failure() { let recorder = Arc::new(FailingRecorder::never_failing()); - let broker = ScriptedBroker::replying().listing(Listing::Fails(CompletionErrorKind::Transport)); + let kind = CompletionErrorKind::Transport; + let broker = ScriptedBroker::replying().listing(Listing::Fails(kind)); - let error = harness_over(&recorder, broker) + let report = harness_over(&recorder, broker) .run(request(PLAIN)) .await - .expect_err("a failed listing is the run's error"); - assert!( - matches!( - error, - HarnessError::Model(CurrentModelError::CatalogFetchFailed(_)) - ), - "{error:?}" + .expect("a failed listing is the run's outcome"); + let lookup = CurrentModelError::CatalogFetchFailed(CompletionError::new(kind, kind.phrase())); + assert_eq!( + report.outcome, + RunOutcome::Failed { + kind: RunErrorKind::Completion, + message: display_chain(&lookup), + }, + "the message is the lookup's cause chain, the broker's message included" ); - assert!( - display_chain(&error).contains(CompletionErrorKind::Transport.phrase()), - "the chain carries the broker's message: {}", - display_chain(&error) + assert_ended_before_the_parse(&recorder, &report); +} + +#[tokio::test] +async fn a_selection_absent_from_the_model_list_ends_the_recorded_run_as_an_unmet_requirement() { + let recorder = Arc::new(FailingRecorder::never_failing()); + let request = RunRequest { + host: HostSnapshot { + selected_model: Some("absent".to_owned()), + ..HostSnapshot::default() + }, + ..request(PLAIN) + }; + + let report = harness_over(&recorder, ScriptedBroker::replying()) + .run(request) + .await + .expect("a selection the list lacks is the run's outcome"); + assert_eq!( + report.outcome, + RunOutcome::Failed { + kind: RunErrorKind::RequirementsUnmet, + message: "the selected model `absent` is absent from the fetched catalog".to_owned(), + } ); - assert_eq!(recorder.calls(), 0, "the run never reached the recorder"); + assert_ended_before_the_parse(&recorder, &report); } #[tokio::test] -async fn a_cancel_while_the_broker_holds_its_list_returns_a_cancelled_report_with_no_run() { +async fn a_cancel_while_the_broker_holds_its_list_ends_the_recorded_run_cancelled() { let recorder = Arc::new(FailingRecorder::never_failing()); let broker = ScriptedBroker::replying().listing(Listing::Holds); let listings = broker.listings(); @@ -93,15 +136,13 @@ async fn a_cancel_while_the_broker_holds_its_list_returns_a_cancelled_report_wit }, ) .await - .expect("a cancel before the run begins is an outcome"); - assert_eq!(report.run_id, None, "no run began"); + .expect("a cancel during the model lookup is an outcome"); assert_eq!(report.outcome, RunOutcome::Cancelled); - assert_eq!(report.output, Err(OutputError::NotCompleted)); - assert_eq!(recorder.calls(), 0, "the run never reached the recorder"); + assert_ended_before_the_parse(&recorder, &report); } #[tokio::test] -async fn a_cancel_before_run_returns_a_cancelled_report_with_no_run_against_a_ready_listing() { +async fn a_cancel_before_run_ends_the_recorded_run_cancelled_against_a_ready_listing() { let recorder = Arc::new(FailingRecorder::never_failing()); let harness = harness_over(&recorder, ScriptedBroker::replying()); harness.control().cancel(); @@ -110,10 +151,8 @@ async fn a_cancel_before_run_returns_a_cancelled_report_with_no_run_against_a_re .run(request(PLAIN)) .await .expect("a cancel before the run begins is an outcome"); - assert_eq!(report.run_id, None, "no run began"); assert_eq!(report.outcome, RunOutcome::Cancelled); - assert_eq!(report.output, Err(OutputError::NotCompleted)); - assert_eq!(recorder.calls(), 0, "the run never reached the recorder"); + assert_ended_before_the_parse(&recorder, &report); } #[tokio::test] @@ -128,6 +167,25 @@ async fn a_recorder_failure_returns_the_recorder_error_holding_the_issued_run_id }; assert_eq!(run, None, "a refused begin issued no run"); + // A run that ends before its prompt is parsed has one write after its + // begin: its end. + let refuses_end = Arc::new(FailingRecorder::failing_on(2)); + let harness = harness_over(&refuses_end, ScriptedBroker::replying()); + harness.control().cancel(); + let error = harness + .run(request(STORES)) + .await + .expect_err("a refused end is the run's error"); + let HarnessError::Recorder { run, .. } = error else { + panic!("the refusal is the recorder's: {error:?}"); + }; + assert_eq!( + run, + Some(refuses_end.begun()[0]), + "the error holds the issued run id" + ); + assert_eq!(refuses_end.inner().outcome(refuses_end.begun()[0]), None); + // Every later call names the run the recorder issued: the parse's // events in preparation, the loop's records, and the run's end. let baseline = Arc::new(FailingRecorder::never_failing()); @@ -190,7 +248,7 @@ async fn a_panicking_broker_answers_its_round_dropped() { }, "a panic in the {what}: the pcall caught the dropped round" ); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); assert_eq!( answers_to(&records, "Chat"), [serde_json::json!("Dropped")], diff --git a/crates/harness-internal/runner/tests/it/harness-stop-cap.rs b/crates/harness-internal/runner/tests/it/harness-stop-cap.rs index 62ffcf63d..a36d0e59e 100644 --- a/crates/harness-internal/runner/tests/it/harness-stop-cap.rs +++ b/crates/harness-internal/runner/tests/it/harness-stop-cap.rs @@ -37,8 +37,8 @@ fn loops_offering(tools: &[&str]) -> String { } format!( "---\nname: loops\ndescription: d\nrunvil: 0\nuses:\n{packs}\ - models:\n writer: {{}}\n---\n\n\ - # Loops\n\n```lua\nmodels.default('writer')\n```\n\n\ + ---\n\n\ + # Loops\n\n\ ## Only\n\n```lua\n{offers}messages:user('hold')\n\ local ok, err = pcall(models.loop)\n\ return tostring(ok) .. ':' .. err.kind\n```\n" @@ -77,8 +77,8 @@ async fn settle() { /// Main spawns Child, which asks the operator; Main then waits on a /// `wait` round, cancels Child, and waits on an `after` round. const CANCELS_AN_ASKER: &str = "---\nname: cancels\ndescription: d\nrunvil: 0\n\ - uses:\n - tests/user-input\nmodels:\n writer: {}\n---\n\n\ - # Cancels\n\n```lua\nmodels.default('writer')\n```\n\n\ + uses:\n - tests/user-input\n---\n\n\ + # Cancels\n\n\ ## Main\n\n```lua\n\ local t = tasks.spawn('## Child')\n\ models.infer('wait')\n\ @@ -144,7 +144,7 @@ async fn a_stop_drops_waiting_calls_without_starting_them() { ); assert_eq!(held.started(), 10, "the waiting call never started"); assert_eq!(held.dropped(), 10, "the stop tore down the ten in flight"); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); assert_eq!( answers_to(&records, "ToolCall"), vec![json!("Dropped"); 11], @@ -200,7 +200,7 @@ async fn a_stop_starts_a_waiting_call_that_survives_stops() { ); assert_eq!(held.started(), 10); assert_eq!(held.dropped(), 10, "the stop tore down the ten holds"); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); assert_eq!( answers_where(&records, "ToolCall", calls(HOLD_TOOL)), vec![json!("Dropped"); 10] @@ -241,7 +241,7 @@ async fn cancelling_a_task_drops_its_question_to_the_operator() { 1, "cancelling the task tore its question down" ); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); assert_eq!( answers_where(&records, "ToolCall", calls(ASK_TOOL)), vec![json!("Dropped")], diff --git a/crates/harness-internal/runner/tests/it/harness-stop-timing.rs b/crates/harness-internal/runner/tests/it/harness-stop-timing.rs index ff3e793ff..063236b70 100644 --- a/crates/harness-internal/runner/tests/it/harness-stop-timing.rs +++ b/crates/harness-internal/runner/tests/it/harness-stop-timing.rs @@ -27,8 +27,8 @@ use crate::support::registry; /// ends, the main infers once, joins the child for the operator's answer, /// and returns how the wait ended, the reply, and the answer. const STOPS_TIMED_JOIN: &str = "---\nname: timed\ndescription: d\nrunvil: 0\n\ - uses:\n - tests/user-input\nmodels:\n writer: {}\n---\n\n\ - # Timed\n\n```lua\nmodels.default('writer')\n```\n\n\ + uses:\n - tests/user-input\n---\n\n\ + # Timed\n\n\ ## Main\n\n```lua\n\ local asker = tasks.spawn('## Asker')\n\ local ok, err = pcall(tasks.join_any, { asker }, { timeout = 30 })\n\ @@ -41,8 +41,8 @@ const STOPS_TIMED_JOIN: &str = "---\nname: timed\ndescription: d\nrunvil: 0\n\ /// round, writes a store file, and then only waits on the child, returning /// the child's report of how its round ended. const STOPS_BEFORE_A_STEP: &str = "---\nname: before-step\ndescription: d\nrunvil: 0\n\ - models:\n writer: {}\n---\n\n\ - # BeforeStep\n\n```lua\nmodels.default('writer')\n```\n\n\ + ---\n\n\ + # BeforeStep\n\n\ ## Main\n\n```lua\n\ local child = tasks.spawn('## Child')\n\ store.write('note.md', 'kept')\n\ @@ -54,14 +54,14 @@ const STOPS_BEFORE_A_STEP: &str = "---\nname: before-step\ndescription: d\nrunvi /// A prompt that hands the operator's answer to one model round. const ASKS_THEN_INFERS: &str = "---\nname: asks-infers\ndescription: d\nrunvil: 0\n\ - uses:\n - tests/user-input\nmodels:\n writer: {}\n---\n\n\ - # AsksInfers\n\n```lua\nmodels.default('writer')\n```\n\n\ + uses:\n - tests/user-input\n---\n\n\ + # AsksInfers\n\n\ ## Only\n\n```lua\nreturn models.infer((input.ask()))\n```\n"; /// A prompt that writes a store file, then runs one model round. const WRITES_THEN_INFERS: &str = "---\nname: writes-infers\ndescription: d\nrunvil: 0\n\ - models:\n writer: {}\n---\n\n\ - # WritesInfers\n\n```lua\nmodels.default('writer')\n```\n\n\ + ---\n\n\ + # WritesInfers\n\n\ ## Only\n\n```lua\nstore.write('note.md', 'kept')\nreturn models.infer('after')\n```\n"; /// A Host that built only the fixture ask Pack, as `user-input`, beside @@ -125,7 +125,7 @@ async fn a_stop_interrupts_a_timed_join_any_and_its_pcall_resumes_while_the_join 0, "the stop left the child's question open" ); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); assert_eq!(answers_to(&records, "Timer"), [json!("Dropped")]); let asks = answers_where(&records, "ToolCall", calls("user-input/ask")); assert_eq!(asks.len(), 1, "one question: {asks:?}"); @@ -168,7 +168,7 @@ async fn a_stop_seen_before_a_steps_effects_start_drops_the_round_in_flight_and_ }, "the child's pcall caught its dropped round, and the main joined the child" ); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); assert_eq!( answers_to(&records, "Chat"), [json!("Dropped")], @@ -218,7 +218,7 @@ async fn a_stop_with_only_a_question_open_spares_the_round_after_its_answer() { "the round reached the broker" ); assert_eq!(operator.abandoned.load(Ordering::SeqCst), 0); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); let chats = answers_to(&records, "Chat"); assert_eq!(chats.len(), 1, "one round: {chats:?}"); assert_ne!( @@ -257,7 +257,7 @@ async fn a_stop_raised_with_nothing_in_flight_changes_nothing_for_the_next_effec 1, "the round reached the broker" ); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); let chats = answers_to(&records, "Chat"); assert_eq!(chats.len(), 1, "one round: {chats:?}"); assert_ne!( diff --git a/crates/harness-internal/runner/tests/it/harness-stop.rs b/crates/harness-internal/runner/tests/it/harness-stop.rs index d1a5d106f..eba50f764 100644 --- a/crates/harness-internal/runner/tests/it/harness-stop.rs +++ b/crates/harness-internal/runner/tests/it/harness-stop.rs @@ -33,8 +33,8 @@ const ASK_TOOL: &str = "user-input/ask"; /// then joins both children and returns how its own wait ended beside /// both children's reports. const STOPS_ALL: &str = "---\nname: stops\ndescription: d\nrunvil: 0\n\ - uses:\n - tests/harness\nmodels:\n writer: {}\n---\n\n\ - # Stops\n\n```lua\nmodels.default('writer')\n```\n\n\ + uses:\n - tests/harness\n---\n\n\ + # Stops\n\n\ ## Main\n\n```lua\n\ local chat = tasks.spawn('## Chat')\n\ local tool = tasks.spawn('## Tool')\n\ @@ -50,16 +50,16 @@ const STOPS_ALL: &str = "---\nname: stops\ndescription: d\nrunvil: 0\n\ /// A prompt whose one round sits under nothing that catches. const STOPS_UNCAUGHT: &str = "---\nname: uncaught\ndescription: d\nrunvil: 0\n\ - models:\n writer: {}\n---\n\n\ - # Uncaught\n\n```lua\nmodels.default('writer')\n```\n\n\ + ---\n\n\ + # Uncaught\n\n\ ## Only\n\n```lua\nreturn models.infer('held')\n```\n"; /// The built-in chat's shape: ask the operator, run the model loop under /// a `pcall`, and ask again, with the held tool advertised to the model. const CHATS: &str = "---\nname: chats\ndescription: d\nrunvil: 0\n\ uses:\n - tests/harness\n - tests/user-input\n\ - models:\n writer: {}\n---\n\n\ - # Chats\n\n```lua\nmodels.default('writer')\n```\n\n\ + ---\n\n\ + # Chats\n\n\ ## Conversation\n\n```lua\n\ tools.offer('harness/hold')\n\ while true do\n\ @@ -72,8 +72,8 @@ const CHATS: &str = "---\nname: chats\ndescription: d\nrunvil: 0\n\ /// how the loop call ended. const LOOPS_ONCE: &str = "---\nname: loops\ndescription: d\nrunvil: 0\n\ uses:\n - tests/harness\n\ - models:\n writer: {}\n---\n\n\ - # Loops\n\n```lua\nmodels.default('writer')\n```\n\n\ + ---\n\n\ + # Loops\n\n\ ## Only\n\n```lua\n\ tools.offer('harness/hold')\n\ messages:user('hold twice')\n\ @@ -128,7 +128,7 @@ async fn a_stop_drops_a_chat_round_a_tool_call_and_a_timer_and_a_pcall_keeps_the assert_eq!(held.started(), 1, "the {what} started once"); assert_eq!(held.dropped(), 1, "the stopped {what} was torn down"); } - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); for kind in ["Chat", "ToolCall", "Timer"] { assert_eq!( answers_to(&records, kind), @@ -159,7 +159,7 @@ async fn an_uncaught_stopped_round_ends_the_run_cancelled() { .expect("the run reaches an outcome"); assert_eq!(report.outcome, RunOutcome::Cancelled); - let run_id = report.run_id.expect("the run began"); + let run_id = report.run_id; assert_eq!(recorder.outcome(run_id), Some(RunOutcome::Cancelled)); assert_eq!( answers_to(&recorder.records(run_id), "Chat"), @@ -227,7 +227,7 @@ async fn two_calls_in_one_round_are_in_flight_at_once_and_a_stop_drops_both() { "the loop's pcall caught the dropped batch" ); assert_eq!(held.dropped(), 2, "the stop tore both calls down"); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); assert_eq!( answers_to(&records, "ToolCall"), [json!("Dropped"), json!("Dropped")], @@ -281,7 +281,7 @@ async fn a_chat_shaped_prompt_returns_to_its_question_after_a_stop_during_a_tool 1, "the cancel tore down the second question" ); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); assert_eq!( answers_where(&records, "ToolCall", calls("harness/hold")), [json!("Dropped")], @@ -346,7 +346,7 @@ async fn a_stop_leaves_a_question_to_the_operator_open() { }, "the question survived the stop and its answer ended the run" ); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); assert_eq!(answers_to(&records, "ToolCall").len(), 1); assert_ne!(answers_to(&records, "ToolCall")[0], json!("Dropped")); } @@ -410,6 +410,6 @@ async fn a_cancel_drops_a_question_to_the_operator() { 1, "the cancel tore the question down" ); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); assert_eq!(answers_to(&records, "ToolCall"), [json!("Dropped")]); } diff --git a/crates/harness-internal/runner/tests/it/harness.rs b/crates/harness-internal/runner/tests/it/harness.rs index 96d61aafa..aaef7ff1f 100644 --- a/crates/harness-internal/runner/tests/it/harness.rs +++ b/crates/harness-internal/runner/tests/it/harness.rs @@ -1,14 +1,15 @@ -//! The per-run Harness: `run` resolves the launch model through the -//! Host's broker, prepares the request's source, drives the run as one -//! future, and reads the declared output; source that fails to parse is a -//! recorded failed run; a Vfs effect is answered inline as it is issued; -//! and one run completes under a thread-parking executor with no runtime -//! behind it. How a stop and a cancel reach the effects in flight sits in -//! the `stop` child module, how a stop reaches the tool calls waiting for -//! a slot and how a task's early end drops its stop-surviving call while -//! the run goes on in the `stop_cap` child module, how what a stop -//! reaches depends on when the loop sees it in the `stop_timing` child -//! module, and every error `run` returns in the `errors` child module. +//! The per-run Harness: `run` begins the run's record, resolves the +//! launch model through the Host's broker, prepares the request's source, +//! drives the run as one future, and reads the declared output; source +//! that fails to parse is a recorded failed run; a Vfs effect is answered +//! inline as it is issued; and one run completes under a thread-parking +//! executor with no runtime behind it. How a stop and a cancel reach the +//! effects in flight sits in the `stop` child module, how a stop reaches +//! the tool calls waiting for a slot and how a task's early end drops its +//! stop-surviving call while the run goes on in the `stop_cap` child +//! module, how what a stop reaches depends on when the loop sees it in the +//! `stop_timing` child module, and the runs that end before the prompt is +//! parsed and the one error `run` returns in the `errors` child module. use std::future::Future; use std::pin::Pin; @@ -19,7 +20,7 @@ use std::time::Duration; use harness_runner::environment::HostSnapshot; use harness_runner::files::OutputError; -use harness_runner::recorder::{MemoryRecorder, Record, RecordKind, RunOutcome}; +use harness_runner::recorder::{MemoryRecorder, Record, RecordKind, RunErrorKind, RunOutcome}; use harness_runner::{Harness, HarnessError, HostContext, RunControl, RunReport, RunRequest}; use runvil::vfs::VfsRef; use runvil_plugin::{HostServices, PackRegistry}; @@ -156,8 +157,8 @@ const WRITES_TWICE: &str = "---\nname: writes\ndescription: d\nrunvil: 0\n---\n\ /// A prompt that writes a store file, infers once through its one role, /// and returns the reply beside the file. const INFERS_AND_STORES: &str = "---\nname: infers\ndescription: d\nrunvil: 0\n\ - models:\n writer: {}\n---\n\n\ - # Title\n\n```lua\nmodels.default('writer')\n```\n\n## Only\n\n```lua\n\ + ---\n\n\ + # Title\n\n## Only\n\n```lua\n\ store.write('note.md', 'kept')\n\ return models.infer('asked') .. '|' .. store.read('note.md')\n```\n"; @@ -179,7 +180,7 @@ async fn run_returns_the_declared_output_and_the_outcome_under_the_requests_name } ); assert_eq!(report.output, Ok("seen: # Paper".to_owned())); - let run_id = report.run_id.expect("the run began at the recorder"); + let run_id = report.run_id; assert_eq!( recorder.outcome(run_id), Some(report.outcome.clone()), @@ -233,13 +234,11 @@ async fn source_that_fails_to_parse_ends_as_a_recorded_failed_run_with_its_parse .await .expect("a parse failure is the run's outcome, not an error"); - let run_id = report - .run_id - .expect("the run began before its source parsed"); + let run_id = report.run_id; let RunOutcome::Failed { kind, .. } = &report.outcome else { panic!("the run failed: {:?}", report.outcome); }; - assert_eq!(kind, "Parse"); + assert_eq!(*kind, RunErrorKind::Parse); assert_eq!(recorder.outcome(run_id), Some(report.outcome.clone())); assert_eq!(report.output, Err(OutputError::NotCompleted)); let records = recorder.records(run_id); @@ -289,7 +288,7 @@ async fn a_vfs_effect_is_answered_inline_on_the_thread_that_polls_the_run() { vec![thread::current().id(); 2], "both writes ran inline, on the thread that polls the run" ); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); for (position, effect) in records.iter().enumerate() { if effect.kind == RecordKind::Effect { let answer = &records[position + 1]; diff --git a/crates/harness-internal/runner/tests/it/performers.rs b/crates/harness-internal/runner/tests/it/performers.rs index ec47933fd..83543d197 100644 --- a/crates/harness-internal/runner/tests/it/performers.rs +++ b/crates/harness-internal/runner/tests/it/performers.rs @@ -19,7 +19,8 @@ use runvil::vfs::Access; use serde_json::{Value, json}; use crate::support::{ - PendingTool, RaiseOnDrop, TEST_SEED, TIMED_MAIN, WAITS, run, run_with_child, unused, + PendingTool, RaiseOnDrop, TEST_SEED, TIMED_MAIN, WAITS, run, run_with_child, + run_with_child_and_cancel, unused, }; /// A Host's timer: sleeps on the test runtime's clock for the effect's @@ -212,7 +213,7 @@ async fn a_pending_timer_is_torn_down_by_a_cancel() { // run must end at the cancel, not when the timer would have fired. let started = Instant::now(); let outcome = drive_run( - run_with_child(TIMED_MAIN, WAITS), + run_with_child_and_cancel(TIMED_MAIN, WAITS, &cancel), performers, recorder.clone(), run_id, diff --git a/crates/harness-internal/runner/tests/it/prepare-files.rs b/crates/harness-internal/runner/tests/it/prepare-files.rs index c0bf57cea..7c30ccf33 100644 --- a/crates/harness-internal/runner/tests/it/prepare-files.rs +++ b/crates/harness-internal/runner/tests/it/prepare-files.rs @@ -1,9 +1,10 @@ //! The prompt's declared store files: the launch's input text is staged //! at the frontmatter's `input:` path before the run, through the store //! view wherever the handle mounts the store; a Host-seeded store -//! satisfies the declaration; text with no declared file, a declared file -//! with neither text nor a seeded copy, and a store that refuses the write -//! each refuse the run and end it under the `Input` kind; and the +//! satisfies the declaration; text with no declared file and a declared +//! file with neither text nor a seeded copy each refuse the run and end it +//! under the `RequirementsUnmet` kind, and a store that refuses the write +//! ends it under the `Vfs` kind; and the //! declared `output:` path comes back on the prepared run for the caller //! to read once the run completes. @@ -12,13 +13,13 @@ use std::sync::Arc; use harness_runner::display_chain; use harness_runner::effect_loop::drive_run; use harness_runner::files::{InputFileError, read_output}; -use harness_runner::prepare::{PrepareError, Prepared, prepare}; -use harness_runner::recorder::{MemoryRecorder, RunOutcome}; +use harness_runner::prepare::{PrepareError, Prepared}; +use harness_runner::recorder::{MemoryRecorder, RunErrorKind, RunOutcome}; use runvil::cancel::CancelHandle; use runvil::event::Event; use runvil::vfs::{MemoryBackend, Mode, ModePolicy, Origin, VfsError, VfsRef}; -use super::{PLAIN, bare, completed, recorded_parse_events, recorder, services}; +use super::{PLAIN, bare, begin_and_prepare, completed, recorded_parse_events, recorder, services}; /// A prompt declaring `paper.md` as its input and `report.md` as its /// output, whose one section writes the output from the input. @@ -38,24 +39,24 @@ async fn prepare_over( let mut services = services(&recorder, bare()); services.vfs = vfs.clone(); services.input_text = input_text.map(str::to_owned); - let prepared = prepare(source, "", services).await; + let prepared = begin_and_prepare(source, services).await; (recorder, prepared) } -/// Checks that `error` is an input refusal whose run ended under the -/// `Input` kind with the cause chain as its message, and that the -/// recorder holds only the run's parse events. Returns the cause. -fn refused(recorder: &MemoryRecorder, error: PrepareError) -> InputFileError { +/// Checks that `error` is an input refusal whose run ended under `kind` +/// with the cause chain as its message, and that the recorder holds only +/// the run's parse events. Returns the cause. +fn refused(recorder: &MemoryRecorder, error: PrepareError, kind: RunErrorKind) -> InputFileError { let PrepareError::Input { run_id, source, .. } = error else { panic!("the failure is an input refusal: {error}"); }; assert_eq!( recorder.outcome(run_id), Some(RunOutcome::Failed { - kind: "Input".to_owned(), + kind, message: display_chain(&source), }), - "the recorder holds the refusal under the Input kind" + "the recorder holds the refusal under the {kind:?} kind" ); let events = recorded_parse_events(recorder, run_id); assert!( @@ -114,14 +115,22 @@ async fn a_host_seeded_store_satisfies_the_declared_input() { #[tokio::test] async fn input_text_for_a_prompt_that_declares_no_input_is_refused() { let (recorder, prepared) = prepare_over(PLAIN, &VfsRef::default(), Some("# Paper")).await; - let source = refused(&recorder, prepared.expect_err("undeclared input refuses")); + let source = refused( + &recorder, + prepared.expect_err("undeclared input refuses"), + RunErrorKind::RequirementsUnmet, + ); assert!(matches!(source, InputFileError::Undeclared), "{source:?}"); } #[tokio::test] async fn a_declared_input_neither_supplied_nor_in_the_store_is_refused() { let (recorder, prepared) = prepare_over(FILES, &VfsRef::default(), None).await; - let source = refused(&recorder, prepared.expect_err("a missing input refuses")); + let source = refused( + &recorder, + prepared.expect_err("a missing input refuses"), + RunErrorKind::RequirementsUnmet, + ); let InputFileError::Missing { path } = source else { panic!("the refusal names the missing file: {source:?}"); }; @@ -135,7 +144,11 @@ async fn a_store_that_refuses_the_staging_write_refuses_the_run() { .policy(ModePolicy::new(Mode::Ask)) .build(); let (recorder, prepared) = prepare_over(FILES, &vfs, Some("# Paper")).await; - let source = refused(&recorder, prepared.expect_err("a refused write refuses")); + let source = refused( + &recorder, + prepared.expect_err("a refused write refuses"), + RunErrorKind::Vfs, + ); let InputFileError::Vfs { path, source } = source else { panic!("the refusal is the store's: {source:?}"); }; diff --git a/crates/harness-internal/runner/tests/it/prepare-host-services.rs b/crates/harness-internal/runner/tests/it/prepare-host-services.rs index e9d17e514..c92daad21 100644 --- a/crates/harness-internal/runner/tests/it/prepare-host-services.rs +++ b/crates/harness-internal/runner/tests/it/prepare-host-services.rs @@ -1,6 +1,7 @@ //! The run's services at preparation: a declared Pack that needs a //! service serves a call with the provider the run's map holds, and a run -//! whose services lack it is refused naming the service. +//! whose services lack it is refused at its first step naming the +//! service. use super::*; @@ -60,32 +61,22 @@ impl Pack for Greeter { } } -/// Prepares [`GREETS`] with `run` as the run's services. -async fn prepare_greeter( - run: HostServices, -) -> (Arc, Result) { +/// Prepares [`GREETS`] with `run` as the run's services and drives the +/// run to its end, returning the recorder and the run's id beside its +/// outcome. +async fn drive_greeter(run: HostServices) -> (Arc, RunId, RunOutcome) { let recorder = recorder(); let mut services = services(&recorder, building(GREETER)); services.services = run; - let prepared = prepare(GREETS, "", services).await; - (recorder, prepared) + let (run_id, outcome) = prepare_and_drive(GREETS, services).await; + (recorder, run_id, outcome) } #[tokio::test] async fn a_pack_serves_a_call_with_the_service_the_runs_map_provides() { let mut run = HostServices::new(); run.provide(&GREETING, Arc::from("hello")).unwrap(); - let (recorder, prepared) = prepare_greeter(run).await; - let prepared = prepared.expect("the run's service meets the need"); - let outcome = drive_run( - prepared.run, - prepared.performers, - recorder.clone(), - prepared.run_id, - CancelHandle::new(), - ) - .await - .unwrap(); + let (_recorder, _run_id, outcome) = drive_greeter(run).await; assert_eq!( completed(outcome), "hello", @@ -95,15 +86,13 @@ async fn a_pack_serves_a_call_with_the_service_the_runs_map_provides() { #[tokio::test] async fn a_pack_whose_service_the_run_lacks_is_refused_naming_the_service() { - let (recorder, prepared) = prepare_greeter(HostServices::new()).await; - let Err(PrepareError::Refused { run_id, error }) = prepared else { - panic!("a required Pack without its service refuses the run"); - }; + let (recorder, run_id, outcome) = drive_greeter(HostServices::new()).await; + let notice = refusal(outcome); assert!( - error - .to_string() - .contains("- greeter needs tests/greeting, and the environment provides none"), - "the notice names the Pack and the missing service: {error}" + notice.contains( + "- greeter is unavailable: the run lacks the services it needs: tests/greeting" + ), + "the notice names the Pack and the missing service: {notice}" ); let effects = recorder .records(run_id) diff --git a/crates/harness-internal/runner/tests/it/prepare-input.rs b/crates/harness-internal/runner/tests/it/prepare-input.rs index 36472531a..0013a9b51 100644 --- a/crates/harness-internal/runner/tests/it/prepare-input.rs +++ b/crates/harness-internal/runner/tests/it/prepare-input.rs @@ -88,19 +88,7 @@ fn with_input(input: Option>) -> HostServices { /// Prepares `source` under `services` and drives the run to its end. async fn drive_prompt(source: &str, services: Services) -> RunOutcome { - let recorder = Arc::clone(&services.recorder); - let prepared = prepare(source, "", services) - .await - .expect("the prompt prepares"); - drive_run( - prepared.run, - prepared.performers, - recorder, - prepared.run_id, - CancelHandle::new(), - ) - .await - .expect("the loop reaches an outcome") + prepare_and_drive(source, services).await.1 } /// What the probe's one call saw, on a run with `input` as its broker. @@ -156,20 +144,18 @@ fn user_input_services( #[tokio::test] async fn a_required_user_input_declaration_on_a_host_without_a_broker_is_refused() { let recorder = recorder(); - let error = prepare( - &user_input_prompt(REQUIRED, ASKS_ONCE), - "", - user_input_services(&recorder, None), - ) - .await - .expect_err("a required Pack without its service refuses the run"); - let PrepareError::Refused { error, .. } = error else { - panic!("the refusal is a requirements refusal: {error}"); - }; - assert_eq!(error.kind(), RunErrorKind::RequirementsUnmet); - let notice = error.to_string(); + let notice = refusal( + drive_prompt( + &user_input_prompt(REQUIRED, ASKS_ONCE), + user_input_services(&recorder, None), + ) + .await, + ); assert!( - notice.contains("- user-input needs tests/input-broker, and the environment provides none"), + notice.contains( + "- user-input is unavailable: the run lacks the services it needs: \ + tests/input-broker" + ), "the notice names the Pack and the missing service: {notice}" ); assert!( @@ -211,7 +197,7 @@ async fn an_uncaught_failed_ask_ends_the_run_as_a_tool_failure() { let RunOutcome::Failed { kind, message } = outcome else { panic!("an uncaught failed ask fails the run: {outcome:?}"); }; - assert_eq!(kind, "Tool"); + assert_eq!(kind, RunErrorKind::Tool); assert!( message.contains("the operator's window closed"), "the broker's message reaches the run's failure: {message}" diff --git a/crates/harness-internal/runner/tests/it/prepare-prelude.rs b/crates/harness-internal/runner/tests/it/prepare-prelude.rs index 036ea9c77..64616cad1 100644 --- a/crates/harness-internal/runner/tests/it/prepare-prelude.rs +++ b/crates/harness-internal/runner/tests/it/prepare-prelude.rs @@ -46,7 +46,7 @@ fn script_call(value: &str, section: &str, entry: &str) -> serde_json::Value { #[tokio::test] async fn a_preludes_tool_calls_are_recorded_as_script_calls_from_the_section_that_made_each() { let recorder = recorder(); - let prepared = prepare(SPEAKS, "", services(&recorder, building(SPEAKER))) + let prepared = begin_and_prepare(SPEAKS, services(&recorder, building(SPEAKER))) .await .unwrap(); let run_id = prepared.run_id; @@ -91,7 +91,7 @@ async fn preludes_run_in_the_prompts_declaration_order_not_the_registry_order() let RunOutcome::Failed { kind, message } = outcome else { panic!("the two preludes collide: {outcome:?}"); }; - assert_eq!(kind, "Lua"); + assert_eq!(kind, RunErrorKind::Lua); assert!( message.contains( "Pack `first`: its prelude defines the global `shared`, \ diff --git a/crates/harness-internal/runner/tests/it/prepare-ready.rs b/crates/harness-internal/runner/tests/it/prepare-ready.rs index 959b5b5cb..c6dcc259c 100644 --- a/crates/harness-internal/runner/tests/it/prepare-ready.rs +++ b/crates/harness-internal/runner/tests/it/prepare-ready.rs @@ -5,9 +5,10 @@ //! offering it raises its reason, an undeclared third-party Pack that //! never gets ready holds no run up, a built Pack with no tools still //! reaches the prompt's Lua, a cancel during the wait ends the run as -//! cancelled in both the report and the record, a cancel that lands while -//! a refusal is recorded leaves the report holding what the record holds, -//! and a Pack that keeps the default `ready` serves as before. +//! cancelled in both the report and the record, even when the run's first +//! step would refuse it, a cancel that lands while a refusal is recorded +//! leaves the report holding what the record holds, and a Pack that keeps +//! the default `ready` serves as before. use super::*; @@ -211,10 +212,13 @@ async fn a_pack_whose_ready_resolves_late_has_its_tools_in_the_run() { let (host, moves, startup) = slow_host(Phase::Starting); let recorder = recorder(); let (prepared, ()) = tokio::time::timeout(PATIENCE, async { - tokio::join!(prepare(CALLS_SLOW, "", services(&recorder, host)), async { - startup.asked.notified().await; - moves.send(Phase::Ready).unwrap(); - }) + tokio::join!( + begin_and_prepare(CALLS_SLOW, services(&recorder, host)), + async { + startup.asked.notified().await; + moves.send(Phase::Ready).unwrap(); + } + ) }) .await .expect("the Harness waits on the Pack's ready"); @@ -237,7 +241,7 @@ async fn every_starting_pack_is_asked_to_be_ready_before_any_of_them_answers() { let recorder = recorder(); let (prepared, ()) = tokio::time::timeout(PATIENCE, async { tokio::join!( - prepare(COUNTS_OFFERED, "", services(&recorder, host)), + begin_and_prepare(COUNTS_OFFERED, services(&recorder, host)), async { for (_, startup) in &handles { startup.asked.notified().await; @@ -274,25 +278,16 @@ async fn a_declared_pack_whose_ready_fails_refuses_the_run_naming_its_reason() { let reason = "the fixture server refused the handshake"; let (host, _moves, _startup) = slow_host(Phase::Failed(reason)); let recorder = recorder(); - let error = prepare(DECLARES_SLOW, "", services(&recorder, host)) - .await - .expect_err("a declared Pack that will not be ready refuses the run"); - - let PrepareError::Refused { run_id, error } = error else { - panic!("the refusal is a requirements refusal: {error}"); - }; - assert!( - error - .to_string() - .contains(&format!("- slow is unavailable: {reason}")), - "the notice names the Pack and its ready failure: {error}" + let (run_id, outcome) = prepare_and_drive(DECLARES_SLOW, services(&recorder, host)).await; + assert_eq!( + recorder.outcome(run_id).as_ref(), + Some(&outcome), + "the recorder holds the refusal as the run's failure" ); + let notice = refusal(outcome); assert!( - matches!( - recorder.outcome(run_id), - Some(RunOutcome::Failed { kind, .. }) if kind == "RequirementsUnmet" - ), - "the recorder holds the refusal as the run's failure" + notice.contains(&format!("- slow is unavailable: {reason}")), + "the notice names the Pack and its ready failure: {notice}" ); } @@ -383,16 +378,51 @@ async fn a_cancel_while_a_declared_pack_is_still_starting_ends_the_run_cancelled let report = report.expect("a cancelled run is reported"); assert_eq!(report.outcome, RunOutcome::Cancelled); assert_eq!(report.output, Err(OutputError::NotCompleted)); - let run_id = report - .run_id - .expect("the run began at the recorder before the wait"); assert_eq!( - recorder.outcome(run_id), + recorder.outcome(report.run_id), Some(RunOutcome::Cancelled), "the recorder holds the cancel, not a refusal of the unready Pack" ); } +#[tokio::test] +async fn a_refusal_after_a_cancel_cut_the_wait_short_ends_the_run_cancelled_with_no_run_events() { + let (host, _moves, startup) = slow_host(Phase::Starting); + let recorder = recorder(); + let harness = Harness::new( + recorder.clone(), + Arc::new(ScriptedBroker::replying()), + Arc::new(Unused), + Arc::new(host), + HostServices::new(), + ); + let control = harness.control(); + let declares_slow_and_web = "---\nname: slow-and-web\ndescription: d\nrunvil: 0\n\ + uses:\n - slow\n - web\n---\n\n# Title\n\n## Only\n\nDone.\n"; + let (report, ()) = tokio::time::timeout(PATIENCE, async { + tokio::join!(harness.run(request(declares_slow_and_web)), async { + startup.asked.notified().await; + control.cancel(); + }) + }) + .await + .expect("the cancel ends the wait"); + + let report = report.expect("a cancelled run is reported"); + assert_eq!( + report.outcome, + RunOutcome::Cancelled, + "the missing web Pack would refuse the run, but the cancel came first" + ); + let run_id = report.run_id; + assert_eq!(recorder.outcome(run_id), Some(RunOutcome::Cancelled)); + let events = recorded_parse_events(&recorder, run_id); + assert!( + matches!(events.last(), Some(Event::ParseSucceeded { .. })), + "the parse events, then the end, with no run event between: {events:?}" + ); +} + #[tokio::test] async fn a_cancel_that_lands_while_a_refusal_is_recorded_leaves_the_report_holding_the_recorded_failure() { @@ -414,11 +444,15 @@ async fn a_cancel_that_lands_while_a_refusal_is_recorded_leaves_the_report_holdi .await .expect("a refused run ends at once") .expect("a refused run is reported"); - let run_id = report - .run_id - .expect("the run began at the recorder before the refusal"); + let run_id = report.run_id; assert!( - matches!(&report.outcome, RunOutcome::Failed { kind, .. } if kind == "RequirementsUnmet"), + matches!( + report.outcome, + RunOutcome::Failed { + kind: RunErrorKind::RequirementsUnmet, + .. + } + ), "the refusal was decided before the cancel landed: {:?}", report.outcome ); diff --git a/crates/harness-internal/runner/tests/it/prepare.rs b/crates/harness-internal/runner/tests/it/prepare.rs index 686319234..ff3c8ec71 100644 --- a/crates/harness-internal/runner/tests/it/prepare.rs +++ b/crates/harness-internal/runner/tests/it/prepare.rs @@ -1,26 +1,25 @@ //! Run preparation: a prompt whose requirements the environment cannot -//! meet is refused with the Engine's own notice and its run ended as -//! failed; a prompt that does not parse fails the same way under the -//! `Parse` kind, and each failure leaves its parse events at the recorder; -//! each preparation draws a fresh seed and start, both handed to the -//! recorder when the run begins under the run's name; and the prepared tool -//! performer sends a `ToolCall` effect to the Pack its id names. The -//! run's optional input broker - one of its own services, lent to every -//! tool call and behind the fixture ask Pack - sits in the `input` -//! child module, the run's services meeting a Pack's needs sit in the -//! `host_services` child module, a Pack's prelude reaching the prepared -//! run sits in the `prelude` child module, the prompt's declared input -//! and output files sit in the `files` child module, and the wait for -//! each Pack to be ready sits in the `ready` child module. +//! meet is prepared, and its run's first step ends it as failed with the +//! Engine's own notice; a prompt that does not parse fails preparation +//! under the `Parse` kind; each leaves only its parse events at the +//! recorder; each run's begin draws a fresh seed and start, both handed +//! to the recorder under the run's name; and the +//! prepared tool performer sends a `ToolCall` effect to the Pack its id +//! names. The run's optional input broker - one of its own services, lent +//! to every tool call and behind the fixture ask Pack - sits in the +//! `input` child module, the run's services meeting a Pack's needs sit in +//! the `host_services` child module, a Pack's prelude reaching the +//! prepared run sits in the `prelude` child module, the prompt's declared +//! input and output files sit in the `files` child module, and the wait +//! for each Pack to be ready sits in the `ready` child module. use std::sync::Arc; use std::time::{SystemTime, UNIX_EPOCH}; use harness_runner::effect_loop::drive_run; -use harness_runner::prepare::{PrepareError, Prepared, Services, prepare}; -use harness_runner::recorder::{MemoryRecorder, RecordKind, RunId, RunOutcome}; +use harness_runner::prepare::{PrepareError, Prepared, Services, begin, prepare}; +use harness_runner::recorder::{MemoryRecorder, RecordKind, RunErrorKind, RunId, RunOutcome}; use harness_runner::{HostContext, display_chain}; -use runvil::RunErrorKind; use runvil::cancel::CancelHandle; use runvil::event::Event; use runvil_plugin::{ @@ -64,16 +63,16 @@ fn recorder() -> Arc { Arc::new(MemoryRecorder::new()) } -/// The events the recorder holds for `run_id`, the run a failed -/// preparation ended, after asserting they are a parse that reported, in -/// order, and nothing else, because the loop never saw the run. +/// The events the recorder holds for `run_id`, after asserting they are +/// only events and open with the parse. A run that failed preparation, or +/// that its first step refused, records nothing after its parse events. fn recorded_parse_events(recorder: &MemoryRecorder, run_id: RunId) -> Vec { let records = recorder.records(run_id); assert!( records .iter() .all(|record| record.kind == RecordKind::Event), - "a failed preparation records only events" + "a run that never started records only events" ); let events: Vec = records .into_iter() @@ -114,6 +113,18 @@ pub(crate) fn services(recorder: &Arc, host: HostContext) -> Ser } } +/// Begins a run of `source` under the name `services` holds at the +/// recorder it holds, then prepares the run with no arguments. +pub(crate) async fn begin_and_prepare( + source: &str, + services: Services, +) -> Result { + let begun = begin(&*services.recorder, &services.name, source) + .await + .expect("the memory recorder begins the run"); + prepare(source, "", begun, services).await +} + /// The fixture Pack `tools`: one tool, `tools/echo`, echoing its /// `value` argument as trusted text. const TOOLS: PackInfo = echo_pack("tests/tools", None); @@ -185,43 +196,36 @@ fn completed(outcome: RunOutcome) -> String { } #[tokio::test] -async fn an_unmet_requirement_is_refused_with_the_engines_notice_and_its_run_ended_as_failed() { +async fn an_unmet_requirement_ends_the_prepared_run_as_failed_with_the_engines_notice() { let recorder = recorder(); - let error = prepare(NEEDS_WEB, "", services(&recorder, bare())) - .await - .expect_err("a missing required Pack refuses the run"); + let (run_id, outcome) = prepare_and_drive(NEEDS_WEB, services(&recorder, bare())).await; - let PrepareError::Refused { run_id, error, .. } = error else { - panic!("the refusal is a requirements refusal: {error}"); - }; - assert_eq!(error.kind(), RunErrorKind::RequirementsUnmet); let notice = "the environment cannot satisfy this prompt:\n\ - missing required Pack: web"; assert_eq!( - error.to_string(), - notice, - "the refusal is the engine's notice, verbatim" + outcome, + RunOutcome::Failed { + kind: RunErrorKind::RequirementsUnmet, + message: notice.to_owned(), + }, + "the failure is the engine's notice, verbatim" ); - assert_eq!( recorder.outcome(run_id), - Some(RunOutcome::Failed { - kind: "RequirementsUnmet".to_owned(), - message: notice.to_owned(), - }), + Some(outcome), "the recorder holds the refusal as the run's failure" ); let events = recorded_parse_events(&recorder, run_id); assert!( matches!(events.last(), Some(Event::ParseSucceeded { .. })), - "the prompt parsed before the environment refused it: {events:?}" + "the parse events, then the end, with no run event between: {events:?}" ); } #[tokio::test] async fn a_prompt_that_does_not_parse_fails_preparation_and_its_run_ends_as_a_parse_failure() { let recorder = recorder(); - let error = prepare(UNCLOSED, "", services(&recorder, bare())) + let error = begin_and_prepare(UNCLOSED, services(&recorder, bare())) .await .expect_err("a prompt without a closed frontmatter does not parse"); @@ -232,7 +236,7 @@ async fn a_prompt_that_does_not_parse_fails_preparation_and_its_run_ends_as_a_pa assert_eq!( recorder.outcome(run_id), Some(RunOutcome::Failed { - kind: "Parse".to_owned(), + kind: RunErrorKind::Parse, message: display_chain(&source), }), "the recorder holds the parse failure and its cause chain under the Parse kind" @@ -248,10 +252,10 @@ async fn a_prompt_that_does_not_parse_fails_preparation_and_its_run_ends_as_a_pa async fn two_prepared_runs_draw_different_seeds_and_both_begin_at_the_recorder() { let recorder = recorder(); let before = unix_millis_now(); - let first = prepare(PLAIN, "", services(&recorder, bare())) + let first = begin_and_prepare(PLAIN, services(&recorder, bare())) .await .unwrap(); - let second = prepare(PLAIN, "", services(&recorder, bare())) + let second = begin_and_prepare(PLAIN, services(&recorder, bare())) .await .unwrap(); let after = unix_millis_now(); @@ -310,7 +314,7 @@ async fn a_prepared_run_drives_to_its_end_under_its_own_performers() { run_id, performers, .. - } = prepare(PLAIN, "", services(&recorder, bare())) + } = begin_and_prepare(PLAIN, services(&recorder, bare())) .await .unwrap(); let outcome = drive_run( @@ -330,21 +334,41 @@ async fn a_prepared_run_drives_to_its_end_under_its_own_performers() { ); } -/// Prepares `source` over `host` and drives the run to its end. -pub(crate) async fn drive_over(source: &str, host: HostContext) -> RunOutcome { - let recorder = recorder(); - let prepared = prepare(source, "", services(&recorder, host)) - .await - .unwrap(); - drive_run( +/// Prepares `source` with `services` and drives the run to its end, +/// returning the run's id at the recorder beside its outcome. +pub(crate) async fn prepare_and_drive(source: &str, services: Services) -> (RunId, RunOutcome) { + let recorder = Arc::clone(&services.recorder); + let prepared = begin_and_prepare(source, services).await.unwrap(); + let outcome = drive_run( prepared.run, prepared.performers, - recorder.clone(), + recorder, prepared.run_id, CancelHandle::new(), ) .await - .unwrap() + .unwrap(); + (prepared.run_id, outcome) +} + +/// Prepares `source` over `host` and drives the run to its end. +pub(crate) async fn drive_over(source: &str, host: HostContext) -> RunOutcome { + prepare_and_drive(source, services(&recorder(), host)) + .await + .1 +} + +/// The notice of a run its first step refused, after asserting that the +/// outcome is a `RequirementsUnmet` failure. +#[track_caller] +pub(crate) fn refusal(outcome: RunOutcome) -> String { + match outcome { + RunOutcome::Failed { + kind: RunErrorKind::RequirementsUnmet, + message, + } => message, + other => panic!("the run's first step refuses it: {other:?}"), + } } #[tokio::test] @@ -384,20 +408,14 @@ async fn an_undeclared_third_party_pack_is_not_in_the_run() { #[tokio::test] async fn a_bare_entry_another_vendor_answers_is_refused_naming_its_fqn() { - let recorder = recorder(); let declares_bare = "---\nname: bare\ndescription: d\nrunvil: 0\n\ uses:\n - tools\n---\n\n# Title\n\n## Only\n\nDone.\n"; - let error = prepare(declares_bare, "", services(&recorder, building(TOOLS))) - .await - .expect_err("a bare id means the runvil Pack, which the Host lacks"); - let PrepareError::Refused { error, .. } = error else { - panic!("the refusal is a requirements refusal: {error}"); - }; assert_eq!( - error.to_string(), + refusal(drive_over(declares_bare, building(TOOLS)).await), "the environment cannot satisfy this prompt:\n\ - tools is unavailable: tools means runvil/tools, which is not installed; \ - write tests/tools to use that Pack" + write tests/tools to use that Pack", + "a bare id means the runvil Pack, which the Host lacks" ); } diff --git a/crates/harness-internal/runner/tests/it/recorder.rs b/crates/harness-internal/runner/tests/it/recorder.rs index 75a221f88..33effe667 100644 --- a/crates/harness-internal/runner/tests/it/recorder.rs +++ b/crates/harness-internal/runner/tests/it/recorder.rs @@ -8,7 +8,8 @@ use std::sync::Arc; use harness_runner::display_chain; use harness_runner::recorder::{ - MemoryRecorder, Record, RecordKind, RecorderError, RunId, RunMeta, RunOutcome, RunRecorder, + MemoryRecorder, Record, RecordKind, RecorderError, RunErrorKind, RunId, RunMeta, RunOutcome, + RunRecorder, }; use serde_json::json; @@ -242,7 +243,7 @@ async fn a_second_end_run_fails_and_keeps_the_first_outcome() { .end_run( run, RunOutcome::Failed { - kind: "Late".to_owned(), + kind: RunErrorKind::Internal, message: "a second end".to_owned(), }, ) @@ -367,7 +368,7 @@ fn run_outcome_text_names_each_variant() { final_text: String::new(), }; let failed = RunOutcome::Failed { - kind: "Parse".to_owned(), + kind: RunErrorKind::Parse, message: "bad".to_owned(), }; diff --git a/crates/harness-internal/runner/tests/it/render-calls.rs b/crates/harness-internal/runner/tests/it/render-calls.rs index 8f016ab8c..83e6525b1 100644 --- a/crates/harness-internal/runner/tests/it/render-calls.rs +++ b/crates/harness-internal/runner/tests/it/render-calls.rs @@ -125,7 +125,7 @@ const ON_TWO_MODELS: [(&str, &str); 3] = [ ( "OnN", "tools.offer('render/echo')\nmessages:user('hold on n')\n\ - local n = models.get('n')\nlocal ok, err = pcall(n.loop, n)\n\ + local n = models.get('n')\nlocal ok, err = pcall(models.loop, n)\n\ return tostring(ok) .. ':' .. (ok and messages[#messages].content or err.kind)", ), ]; @@ -199,7 +199,7 @@ async fn two_models_get_their_own_renders_and_a_stop_spares_only_the_surviving_c ], "each entry renders for its own round's model" ); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); let mut rendered: Vec = answered(&records, "Render") .into_iter() .map(|(_, answer)| answer["Render"][0]["static_set"].clone()) diff --git a/crates/harness-internal/runner/tests/it/render-entries.rs b/crates/harness-internal/runner/tests/it/render-entries.rs index 762ee5b73..4a39dda3e 100644 --- a/crates/harness-internal/runner/tests/it/render-entries.rs +++ b/crates/harness-internal/runner/tests/it/render-entries.rs @@ -78,7 +78,7 @@ async fn a_panicking_render_answers_dropped_and_the_log_holds_the_render_and_its answered(&records, "Render"), [( json!({ - "alias": "writer", + "alias": "default", "entry": entry, "offered": [{ "pack": "render", "static_set": [ECHO], "dynamic_set": [] }], }), diff --git a/crates/harness-internal/runner/tests/it/render.rs b/crates/harness-internal/runner/tests/it/render.rs index e2cf0100d..2ee593b13 100644 --- a/crates/harness-internal/runner/tests/it/render.rs +++ b/crates/harness-internal/runner/tests/it/render.rs @@ -45,10 +45,10 @@ const SECRET: &str = "render/secret"; const SEEN: ServiceKey = ServiceKey::new("tests/render-seen"); /// The frontmatter and H1 every prompt here shares: the render Pack -/// declared and one role bound as the default. +/// declared and a title. const PREAMBLE: &str = "---\nname: renders\ndescription: d\nrunvil: 0\n\ - uses:\n - tests/render\nmodels:\n writer: {}\n---\n\n\ - # Renders\n\n```lua\nmodels.default('writer')\n```\n\n"; + uses:\n - tests/render\n---\n\n\ + # Renders\n\n"; /// One render as the render Pack saw it. #[derive(Debug, Clone, PartialEq)] @@ -338,10 +338,7 @@ async fn run( RunOutcome::Completed { final_text } => final_text, other => panic!("the run completes: {other:?}"), }; - ( - final_text, - recorder.records(report.run_id.expect("the run began")), - ) + (final_text, recorder.records(report.run_id)) } /// Each effect whose record is keyed by `kind`, as its body beside its diff --git a/crates/harness-internal/runner/tests/it/support.rs b/crates/harness-internal/runner/tests/it/support.rs index 2517548a6..8e667b24f 100644 --- a/crates/harness-internal/runner/tests/it/support.rs +++ b/crates/harness-internal/runner/tests/it/support.rs @@ -2,6 +2,7 @@ //! over it, fake performers that answer by script, and a Pack registry //! holding a suite's Packs. +use std::fmt::Write as _; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; use std::sync::{Arc, Mutex}; use std::task::Poll; @@ -11,6 +12,7 @@ use harness_runner::performers::{BoxFuture, InferenceBroker, Performers, Timer, use harness_runner::recorder::{ MemoryRecorder, Record, RecorderError, RecorderFuture, RunId, RunMeta, RunOutcome, RunRecorder, }; +use runvil::cancel::CancelHandle; use runvil::effect::{Round, ToolCallOrigin}; use runvil::model::{ Completion, CompletionError, CompletionOptions, Message, ModelBinding, ModelCatalog, ToolSchema, @@ -21,7 +23,7 @@ use runvil::vfs::{ Access, AcquireContext, Entry, ExecId, MemoryBackend, Stat, Vfs, VfsAccess, VfsError, VfsPath, VfsRef, }; -use runvil::{Prompt, Run, RunContext, RunSeed}; +use runvil::{Prompt, Run, RunConfig, RunSeed}; use runvil_plugin::{PackInfo, PackRegistry}; use serde_json::{Value, json}; @@ -58,26 +60,27 @@ fn catalog() -> ToolCatalog { ToolCatalog::new([wait]).expect("the one-tool catalog is valid") } -/// A prompt whose one section runs `lua` as its only block. -fn prompt(lua: &str) -> Arc { - let source = format!( - "---\nname: runner-test\ndescription: a runner fixture\nrunvil: 0\n---\n\n\ - # Fixture\n\n## Only\n\n```lua\n{lua}\n```\n" +/// A prompt whose sections are `sections`, each a section's name and the +/// Lua it runs as its only block, in order. +fn prompt(sections: &[(&str, &str)]) -> Arc { + let mut source = String::from( + "---\nname: runner-test\ndescription: a runner fixture\nrunvil: 0\n---\n\n# Fixture\n", ); + for (name, lua) in sections { + write!(source, "\n## {name}\n\n```lua\n{lua}\n```\n").expect("a String takes any write"); + } let (prompt, _parse_events) = Prompt::parse(&source, EXECUTION); Arc::new(prompt.expect("the fixture prompt parses")) } /// A Pack-free run over `prompt` with a fixed seed and start, /// prepared against the fixture [`catalog`], over the run's filesystem -/// `vfs`. -fn prepared(prompt: Arc, vfs: VfsRef) -> Run { - let ctx = RunContext::new(EXECUTION, TEST_SEED, Timestamp::UNIX_EPOCH).with_vfs(vfs); - let (ctx, requirements) = ctx.with_tools(catalog()).prepare(&prompt, None); - assert!( - requirements.is_satisfied(), - "the runner fixture prompt declares nothing the environment must supply: {requirements:?}" - ); +/// `vfs`, under the cancel flag `cancel`. +fn prepared(prompt: Arc, vfs: VfsRef, cancel: &CancelHandle) -> Run { + let ctx = RunConfig::new(EXECUTION, TEST_SEED, Timestamp::UNIX_EPOCH) + .with_vfs(vfs) + .with_tools(catalog()) + .with_cancel(cancel.clone()); Run::new(prompt, "", ctx) } @@ -91,19 +94,26 @@ pub(crate) fn run(lua: &str) -> Run { /// `vfs`: the test holds the handle, so it can read what the run wrote or /// mount a backend of its own. pub(crate) fn run_over(lua: &str, vfs: VfsRef) -> Run { - prepared(prompt(lua), vfs) + run_of_sections(&[("Only", lua)], vfs) +} + +/// A Pack-free run over `sections`, each a section's name and the Lua it +/// runs, whose `store` table operates on `vfs`. +pub(crate) fn run_of_sections(sections: &[(&str, &str)], vfs: VfsRef) -> Run { + prepared(prompt(sections), vfs, &CancelHandle::new()) } /// A Pack-free run over two sections: `## Main` runs `main`, and /// `## Child` runs `child` when the main spawns it as a task. pub(crate) fn run_with_child(main: &str, child: &str) -> Run { - let source = format!( - "---\nname: runner-test\ndescription: a runner fixture\nrunvil: 0\n---\n\n\ - # Fixture\n\n## Main\n\n```lua\n{main}\n```\n\n## Child\n\n```lua\n{child}\n```\n" - ); - let (prompt, _parse_events) = Prompt::parse(&source, EXECUTION); - let prompt = Arc::new(prompt.expect("the two-section fixture prompt parses")); - prepared(prompt, VfsRef::default()) + run_with_child_and_cancel(main, child, &CancelHandle::new()) +} + +/// [`run_with_child`] under the cancel flag `cancel`, which the test also +/// hands the loop. +pub(crate) fn run_with_child_and_cancel(main: &str, child: &str, cancel: &CancelHandle) -> Run { + let sections = [("Main", main), ("Child", child)]; + prepared(prompt(§ions), VfsRef::default(), cancel) } /// A main section that parks on a 30-second timer beside its child: the diff --git a/crates/harness-internal/runner/tests/it/tool_context.rs b/crates/harness-internal/runner/tests/it/tool_context.rs index 6794190f7..b4a9372e3 100644 --- a/crates/harness-internal/runner/tests/it/tool_context.rs +++ b/crates/harness-internal/runner/tests/it/tool_context.rs @@ -6,7 +6,6 @@ use std::sync::Arc; use harness_runner::effect_loop::drive_run; -use harness_runner::prepare::prepare; use harness_runner::recorder::{MemoryRecorder, RunOutcome}; use runvil::cancel::CancelHandle; use runvil_plugin::{ @@ -15,7 +14,7 @@ use runvil_plugin::{ }; use serde_json::{Value, json}; -use crate::prepare::{building, services}; +use crate::prepare::{begin_and_prepare, building, services}; /// What the fixture tool writes to `/from-tool.md`. const WRITTEN: &str = "written through the context"; @@ -77,7 +76,7 @@ impl Pack for Record { #[tokio::test] async fn a_tool_writes_through_its_context_and_the_calling_script_reads_it_back() { let recorder = Arc::new(MemoryRecorder::new()); - let prepared = prepare(CALLS_RECORD, "", services(&recorder, building(CONTEXT))) + let prepared = begin_and_prepare(CALLS_RECORD, services(&recorder, building(CONTEXT))) .await .unwrap(); let outcome = drive_run( diff --git a/crates/harness/examples/run-prompt.rs b/crates/harness/examples/run-prompt.rs index 38a5855c0..0b62376c0 100644 --- a/crates/harness/examples/run-prompt.rs +++ b/crates/harness/examples/run-prompt.rs @@ -87,8 +87,7 @@ async fn main() -> Result<(), Box> { // 4. The report says how the run ended and what it left at its output // file, and the recorder holds the run's outcome. assert!(matches!(report.outcome, RunOutcome::Completed { .. })); - let run_id = report.run_id.ok_or("the run began")?; - assert!(recorder.outcome(run_id).is_some()); + assert!(recorder.outcome(report.run_id).is_some()); let greeting = report.output?; assert_eq!(greeting, "Hello, desk."); println!("{greeting}"); diff --git a/crates/harness/src/lib.rs b/crates/harness/src/lib.rs index d3985848f..d06f3beec 100644 --- a/crates/harness/src/lib.rs +++ b/crates/harness/src/lib.rs @@ -10,6 +10,12 @@ //! goes on, the Host steers it through the [`RunControl`] it took from //! `Harness::control` before the run started. //! +//! Every run gets a record and ends with one outcome: completed, failed +//! with a kind and a message, or cancelled. The record begins before +//! anything else, so a run cancelled before it starts, or one whose model +//! cannot be resolved, is recorded too. The only [`HarnessError`] is a +//! recorder that refuses a write. +//! //! The crate's `run-prompt` example runs one prompt from start to end. pub use harness_runner::Harness; @@ -18,7 +24,6 @@ pub use harness_runner::RunControl; pub use harness_runner::RunReport; pub use harness_runner::RunRequest; pub use harness_runner::display_chain; -pub use harness_runner::environment::CurrentModelError; pub use harness_runner::environment::HostSnapshot; pub use harness_runner::files::OutputError; pub use harness_runner::performers::BoxFuture; @@ -54,13 +59,15 @@ pub mod plugin { pub mod record { //! The recorder that receives every effect, answer, and event of a run, - //! an in-memory recorder, and the seed each run's meta records. + //! an in-memory recorder, the seed each run's meta records, and the + //! Engine's outcome each run ends with, with the kind of a failed one. pub use harness_runner::recorder::MemoryRecorder; pub use harness_runner::recorder::Record; pub use harness_runner::recorder::RecordKind; pub use harness_runner::recorder::RecorderError; pub use harness_runner::recorder::RecorderFuture; + pub use harness_runner::recorder::RunErrorKind; pub use harness_runner::recorder::RunId; pub use harness_runner::recorder::RunMeta; pub use harness_runner::recorder::RunOutcome; diff --git a/crates/harness/tests/suite/broker.rs b/crates/harness/tests/suite/broker.rs index b39eab33e..81cf5ee64 100644 --- a/crates/harness/tests/suite/broker.rs +++ b/crates/harness/tests/suite/broker.rs @@ -25,8 +25,8 @@ const MODEL: &str = "scripted-model"; /// A program that infers once and then chats once: the nested round /// first, then the section's own round over its message list. const INFERS_THEN_CHATS: &str = "---\nname: rounds\ndescription: infers then chats\n\ - runvil: 0\nmodels:\n writer: {}\n---\n\n\ - # Rounds\n\n```lua\nmodels.default('writer')\n```\n\n\ + runvil: 0\n---\n\n\ + # Rounds\n\n\ ## Only\n\n```lua\n\ local inferred = models.infer('infer round')\n\ messages:user('chat round')\n\ diff --git a/crates/harness/tests/suite/host.rs b/crates/harness/tests/suite/host.rs index e7866691e..08ecc7c99 100644 --- a/crates/harness/tests/suite/host.rs +++ b/crates/harness/tests/suite/host.rs @@ -38,10 +38,9 @@ const ASKS: &str = concat!( /// when the round fails. const PATIENT: &str = concat!( "---\nname: patient\ndescription: Waits on the model\nrunvil: 0\n", - "models: { writer: {} }\n", "---\n\n# Patient\n\n## Only\n\n```lua\n", - "local writer = models.get('writer')\n", - "local ok, err = pcall(writer.infer, writer, 'Take your time.')\n", + "local writer = models.get('default')\n", + "local ok, err = pcall(models.infer, writer, 'Take your time.')\n", "return ok and 'answered' or err.kind\n", "```\n", ); @@ -49,10 +48,9 @@ const PATIENT: &str = concat!( /// Runs one section round and returns the model's reply. const CHATS: &str = concat!( "---\nname: chats\ndescription: Says hello to the model\nrunvil: 0\n", - "models: { writer: {} }\n", "---\n\n# Chats\n\n## Only\n\n```lua\n", "messages:user('Hello, desk.')\n", - "models.get('writer'):loop()\n", + "models.loop(models.get('default'))\n", "return messages[#messages].content\n", "```\n", ); @@ -241,15 +239,30 @@ async fn stopping_an_in_flight_round_drops_the_call_and_the_prompts_pcall_catche } #[tokio::test] -async fn a_run_cancelled_before_it_begins_ends_cancelled_and_is_never_recorded() { - let (harness, _round_started) = stuck_harness(stub_catalog().expect("the stub catalog builds")); +async fn a_run_cancelled_before_it_begins_is_recorded_as_cancelled_before_its_parse() { + let recorder = Arc::new(MemoryRecorder::new()); + let harness = Harness::new( + recorder.clone(), + Arc::new(Offline), + Arc::new(Clock), + Arc::new(HostContext::new(PackRegistry::new(), HostServices::new())), + HostServices::new(), + ); harness.control().cancel(); let report = harness .run(request("desk-patient", PATIENT)) .await .expect("the run reaches an outcome"); assert_eq!(report.outcome, RunOutcome::Cancelled); - assert_eq!(report.run_id, None); + assert_eq!( + recorder.outcome(report.run_id), + Some(RunOutcome::Cancelled), + "the recorder holds the cancelled run" + ); + assert!( + recorder.records(report.run_id).is_empty(), + "the run ended before its prompt was parsed" + ); } #[tokio::test] @@ -272,7 +285,7 @@ async fn a_hosts_own_broker_streams_a_section_round_under_the_round_id_the_run_r .await .expect("the run reaches an outcome"); - let records = recorder.records(report.run_id.expect("the run began")); + let records = recorder.records(report.run_id); let reply = records .iter() .filter(|record| record.kind == RecordKind::Event) diff --git a/crates/plugin-mcp/src/result-tests.rs b/crates/plugin-mcp/src/result-tests.rs index 5787c31ea..1e31254b7 100644 --- a/crates/plugin-mcp/src/result-tests.rs +++ b/crates/plugin-mcp/src/result-tests.rs @@ -117,7 +117,6 @@ fn a_timeout_is_a_transport_error_naming_the_deadline() { }); assert_eq!(error.kind(), ToolErrorKind::Transport); assert!(error.to_string().contains("300 seconds")); - assert!(error.is_retryable()); } #[test] diff --git a/crates/plugin-std/AGENTS.md b/crates/plugin-std/AGENTS.md new file mode 100644 index 000000000..d3b8dd25a --- /dev/null +++ b/crates/plugin-std/AGENTS.md @@ -0,0 +1 @@ +- Source code for each tool of this Plugin goes in its own directory named after the Lua name of the tool diff --git a/crates/plugin-std/Cargo.toml b/crates/plugin-std/Cargo.toml index 64137c949..b169ec560 100644 --- a/crates/plugin-std/Cargo.toml +++ b/crates/plugin-std/Cargo.toml @@ -13,6 +13,8 @@ runvil-plugin.workspace = true # Each tool's definition is embedded JSON, read into a spec at construct. serde.workspace = true serde_json.workspace = true +# Grep compiles its pattern with Rust regex syntax. +regex.workspace = true [dev-dependencies] # `testing::TestCall`, which lends the tools a context without a Harness. diff --git a/crates/plugin-std/schemas/EDITS.md b/crates/plugin-std/schemas/EDITS.md index 03e6d744c..1c4ebcbf3 100644 --- a/crates/plugin-std/schemas/EDITS.md +++ b/crates/plugin-std/schemas/EDITS.md @@ -1,9 +1,9 @@ # Definition edits -Each file in this directory is Cursor's definition of the tool named by its -`wire_name`, snapshot on 2026-10-27. Everything in a file is Cursor's text -except the edits below, each made because the line describes behavior this -Pack does not have. +Each file in this directory except `dir.json` is Cursor's definition of the +tool named by its `wire_name`, snapshot on 2026-10-27. Everything in such a +file is Cursor's text except the edits below, each made because the line +describes behavior this Pack does not have. `dir.json` is this crate's own. ## read.json @@ -19,6 +19,25 @@ Pack does not have. - Removed: `Image Support`, `PDF Support`, and the paragraph explaining how images and PDFs are returned. This build reads text only, so a model told otherwise would send a PDF and get an error. +- Removed `Results are returned using cat -n format, with line numbers + starting at 1`. The description now states the format Read really uses: a + six-wide right-aligned number, a pipe, and the line's text, where the + number and the pipe are not part of the line. +- Removed `Any lines longer than 2000 characters will be truncated`. Read + cuts a line only when that line alone passes the byte cap, and the + description says so. +- Added lines that state what Read does: it opens any path the call names + inside what the run has mounted, and reads through a link unless the link + leaves the run's filesystem; a result holds at most `{byte_cap}` bytes of + numbered lines, and a read that stops early names the offset to continue + from; only text is read, so a file over `{file_ceiling}` bytes and a binary + file are refused. The two placeholders are filled when the definition is + built, from the constants the code enforces, so the text and the limits + cannot drift apart. +- The `offset` and `limit` types are `integer`, where Cursor's were + `number`, because a line count is whole. +- The `offset` text now says that 1 is the first line, that 0 reads as 1, and + that a negative value counts back from the end with -1 as the last line. ## write.json @@ -41,13 +60,66 @@ Pack does not have. - The `target_directory` description's default, `Cursor workspace root`, became `the run's filesystem root`. +## dir.json + +- This file is new. It is not a Cursor snapshot, so none of its text is + Cursor's. Cursor's counterpart, `LS`, is deprecated and gated behind a + setting, so the wire name is `ListDir` rather than Cursor's. The two + parameters, `target_directory` and `ignore_globs`, keep the names `LS` + uses. +- The description states what this Pack's tool does: a bullet tree with + directories ending in `/`, one byte-ordered list for files and + directories, a breadth-first walk that stops at `{body_cap}` bytes with a + note, `.git` skipped and every other dotfile listed, links listed and not + entered, and `ignore_globs` matching at any depth. The placeholder is + filled when the definition is built, from the constant the code + enforces, so the text and the limit cannot drift apart. +- Dropped from `LS`: the extension-count summaries, the 5-second timeout, + the terminal-log special case, and the `Error listing directory:` + prefix. + ## grep.json -- Removed the sentence pointing at `rg`, since the tool searches the run's - filesystem itself. -- Removed the sentence pointing at the `Shell` tool for piping, because this - Pack offers no `Shell`. -- The `path` parameter's default, `Cursor workspace root`, became `the - run's filesystem root`. -- Every parameter, type, description, enum, and the `required` list is - otherwise Cursor's. +- The parameter set is now Cursor's bundle schema plus `literal`: `pattern`, + `path`, `glob`, `output_mode`, `-B`, `-A`, `-C`, `-i`, `type`, `head_limit`, + `offset`, `multiline`, and `literal`, in that order. The hyphenated names + are kept and no alias is added. +- Deleted `-n` and `context`. `-n` is gone because Grep always shows line + numbers, and a call that still passes `-n: true` is accepted while any + other `-n` value is refused. `-C` now takes the text `context` had, "Number + of lines to show before and after each match. Requires output_mode: + "content", ignored otherwise.", and `context` is refused as an undeclared + key. +- Added `literal`, a boolean that defaults to false: the pattern is matched as + plain text, so a pattern such as `foo(` needs no escape. +- `output_mode` now defaults to `content`, where the earlier snapshot said + `files_with_matches`. Its text no longer mentions `-n`, and its enum is + unchanged. +- `head_limit` has no default and `"minimum": 0`. Cursor's text said it + defaults to 250 and that 0 is unlimited, and this one says that 0 means no + limit and so does leaving it out. Its text says it counts matching lines in + `content` mode, files in `files_with_matches`, and file rows in `count`. +- `path` reads `File or directory to search in, as an absolute path. Defaults + to the run's filesystem root.`, where Cursor's text said `Cursor workspace + root`, so the earlier edit to that default still holds. +- `glob` reads `Glob pattern to filter files, for example "*.js" or + "**/*.tsx". Only * and ** are special, and matching is case-sensitive. A + pattern that does not start with "/" is matched at any depth. A pattern + that starts with "/" is a full path.`, because `{ts,tsx}` and the other + shell forms are not special in this Pack's matcher. +- The description is new and states what this Pack's Grep does: Rust regex + syntax with no look-around and no back-references, and how to escape or set + `literal`; path order, the order Glob uses; output that stops at + `{body_cap}` bytes; at most 100000 files considered; a file over + `{file_ceiling}` bytes not searched; binary files, symbolic links, over-size + files, and unreadable entries skipped and counted in the result; only + entries named `.git` skipped, with `.gitignore` rules not applied, so + `target/` and `node_modules/` are searched when they lie under `path`; the + `N:` and `N-` line prefixes and that `N` is the number Read shows; a line + over 1000 bytes shown as a window around its first match; and that paths + are absolute and work as the `path` argument of the other file tools. The + two placeholders are filled when the definition is built, from the + constants the code enforces, so the text and the limits cannot drift apart. +- The removals from the earlier snapshot still hold: the sentence that + pointed at `rg` and the sentence that pointed at the `Shell` tool for + piping are gone, and the new text names neither. \ No newline at end of file diff --git a/crates/plugin-std/schemas/dir.json b/crates/plugin-std/schemas/dir.json new file mode 100644 index 000000000..1e74e47e8 --- /dev/null +++ b/crates/plugin-std/schemas/dir.json @@ -0,0 +1,19 @@ +{ + "wire_name": "ListDir", + "description": "Lists a directory of the run's filesystem as a tree. Each entry is a bullet line, `- name`, the entries of a directory are indented two spaces under it, and a directory's name ends in `/`.\n\nUsage:\n- Files and directories share one list, in byte order of the name with `/` added to a directory's name, so `B` comes before `a`.\n- The walk is breadth first, so the entries closest to target_directory come first. The output holds at most {body_cap} bytes of lines. When it stops early, a directory that was not expanded ends in `...`, and a note at the end says the listing is too large. Call again with a more specific target_directory or with ignore_globs to see the rest.\n- Entries named `.git` are skipped. Every other dotfile is listed.\n- A link is listed by its name and is not entered.\n- ignore_globs leave entries out. A pattern with no leading `/` matches at any depth, so `node_modules` or `*.log` leaves out every match below target_directory. A pattern with a leading `/` is a full path from the root of the run's filesystem. Only `*` and `**` are special.", + "parameters": { + "type": "object", + "properties": { + "target_directory": { + "type": "string", + "description": "The absolute path of the directory to list" + }, + "ignore_globs": { + "type": "array", + "items": { "type": "string" }, + "description": "Globs of entries to leave out of the listing. A pattern with no leading `/` matches at any depth, and only `*` and `**` are special." + } + }, + "required": ["target_directory"] + } +} diff --git a/crates/plugin-std/schemas/grep.json b/crates/plugin-std/schemas/grep.json index c874d01b6..1046fd8eb 100644 --- a/crates/plugin-std/schemas/grep.json +++ b/crates/plugin-std/schemas/grep.json @@ -1,6 +1,6 @@ { "wire_name": "Grep", - "description": "Searches file contents with a regular expression and reports the matches. Use it to find where a symbol, string, or pattern occurs, and read the matching file before changing it.", + "description": "Searches file contents with a regular expression and reports the matches. Use it to find where a symbol, string, or pattern occurs, and read the matching file before changing it.\n\nThe pattern uses Rust regex syntax, with no look-around and no back-references. Escape ( [ { . when you mean them literally, or set literal to true to match the pattern as plain text. Matching is case-sensitive unless -i is set.\n\nFiles are searched in path order, the same order Glob uses: directories are entered depth first, and the names in each directory are in byte order. Output stops at {body_cap} bytes, and at most 100000 files are considered in one search. A file over {file_ceiling} bytes is not searched. Binary files, symbolic links, over-size files, and entries that cannot be read are skipped, and the result says how many were skipped and why.\n\nOnly entries named .git are skipped, and .gitignore rules are not applied, so output such as target/ and node_modules/ is searched when it lies under path. Narrow path, glob, or type to keep it out.\n\nIn content mode each match line is shown as N:text and each context line as N-text, under the absolute path of its file. The N and the colon or hyphen are not part of the line, and N is the line number Read shows. A line over 1000 bytes is shown as a window around its first match, with markers that give the bytes left out.\n\nPaths in the result are absolute and work as the path argument of the other file tools.", "parameters": { "type": "object", "properties": { @@ -10,20 +10,17 @@ }, "path": { "type": "string", - "description": "File or directory to search in. Defaults to the run's filesystem root." + "description": "File or directory to search in, as an absolute path. Defaults to the run's filesystem root." }, "glob": { "type": "string", - "description": "Glob pattern to filter files (e.g. \"*.js\", \"*.{ts,tsx}\")" - }, - "type": { - "type": "string", - "description": "File type to search. Common types: js, py, rust, go, java, etc. More efficient than glob for standard file types." + "description": "Glob pattern to filter files, for example \"*.js\" or \"**/*.tsx\". Only * and ** are special, and matching is case-sensitive. A pattern that does not start with \"/\" is matched at any depth. A pattern that starts with \"/\" is a full path." }, "output_mode": { "type": "string", "enum": ["content", "files_with_matches", "count"], - "description": "Output mode: \"content\" shows matching lines (supports -A/-B/-C context, -n line numbers, head_limit), \"files_with_matches\" shows file paths (supports head_limit), \"count\" shows match counts (supports head_limit). Defaults to \"files_with_matches\"." + "default": "content", + "description": "Output mode: \"content\" shows matching lines (supports -A/-B/-C context, head_limit), \"files_with_matches\" shows file paths (supports head_limit), \"count\" shows match counts (supports head_limit). Defaults to \"content\"." }, "-B": { "type": "number", @@ -34,32 +31,34 @@ "description": "Number of lines to show after each match. Requires output_mode: \"content\", ignored otherwise." }, "-C": { - "type": "number", - "description": "Alias for context." - }, - "context": { "type": "number", "description": "Number of lines to show before and after each match. Requires output_mode: \"content\", ignored otherwise." }, - "-n": { - "type": "boolean", - "description": "Show line numbers in output. Requires output_mode: \"content\", ignored otherwise. Defaults to true." - }, "-i": { "type": "boolean", "description": "Case insensitive search" }, - "multiline": { - "type": "boolean", - "description": "Enable multiline mode where . matches newlines and patterns can span lines. Default: false." + "type": { + "type": "string", + "description": "File type to search. Common types: js, py, rust, go, java, etc. More efficient than glob for standard file types." }, "head_limit": { "type": "number", - "description": "Limit output to first N lines/entries. Works across all output modes: content (limits output lines), files_with_matches (limits file paths), count (limits count entries). Defaults to 250 when unspecified. Pass 0 for unlimited (use sparingly - large result sets waste context)." + "minimum": 0, + "description": "Limit output to first N lines/entries. Works across all output modes: content (limits matching lines), files_with_matches (limits file paths), count (limits count entries). Pass 0 for no limit, which is also what leaving it out means." }, "offset": { "type": "number", "description": "Skip first N lines/entries before applying head_limit. Works across all output modes. Defaults to 0." + }, + "multiline": { + "type": "boolean", + "description": "Enable multiline mode where . matches newlines and patterns can span lines. Default: false." + }, + "literal": { + "type": "boolean", + "default": false, + "description": "Match the pattern as plain text instead of as a regular expression, so ( [ { . and the rest need no escape. Default: false." } }, "required": ["pattern"] diff --git a/crates/plugin-std/schemas/read.json b/crates/plugin-std/schemas/read.json index ce3a58158..7f6bdb502 100644 --- a/crates/plugin-std/schemas/read.json +++ b/crates/plugin-std/schemas/read.json @@ -1,6 +1,6 @@ { "wire_name": "Read", - "description": "Reads a file from the run's filesystem. You can access any file directly by using this tool.\nAssume this tool is able to read every file the run can see. If the prompt provides a path to a file assume that path is valid. It is okay to read a file that does not exist; an error will be returned.\n\nUsage:\n- You can optionally specify a line offset and limit (especially handy for long files), but it's recommended to read the whole file by not providing these parameters\n- Any lines longer than 2000 characters will be truncated\n- Results are returned using cat -n format, with line numbers starting at 1\n- You have the capability to call multiple tools in a single response. It is always better to speculatively read multiple files as a batch that are potentially useful.\n- If you read a file that exists but has empty contents you will receive 'File is empty.'", + "description": "Reads a file from the run's filesystem. You can access any file directly by using this tool.\nAssume this tool is able to read every file the run can see. If the prompt provides a path to a file assume that path is valid. It is okay to read a file that does not exist; an error will be returned.\n\nUsage:\n- Read opens any path the call names inside what the run has mounted. A link is read through unless it leaves the run's filesystem.\n- You can optionally specify a line offset and limit (especially handy for long files), but it's recommended to read the whole file by not providing these parameters\n- Each line comes back as a six-wide right-aligned line number, a pipe, and the line's text, like ` 1|first line`. The number and the pipe are not part of the line.\n- A result holds at most {byte_cap} bytes of numbered lines. When a read stops before the last line, its last line names the offset to pass to continue. A single line longer than that is cut, and no parameter reads the rest of it.\n- Only text is read. A file over {file_ceiling} bytes is refused, and so is a binary file.\n- You have the capability to call multiple tools in a single response. It is always better to speculatively read multiple files as a batch that are potentially useful.\n- If you read a file that exists but has empty contents you will receive 'File is empty.'", "parameters": { "type": "object", "properties": { @@ -9,11 +9,11 @@ "description": "The absolute path of the file to read" }, "offset": { - "type": "number", - "description": "The line number to start reading from. Only provide if the file is too large to read at once. Negative values count backwards from the end." + "type": "integer", + "description": "The line number to start reading from. Only provide if the file is too large to read at once. 1 is the first line, 0 reads as 1, and a negative value counts back from the end, so -1 is the last line." }, "limit": { - "type": "number", + "type": "integer", "description": "The number of lines to read. Only provide if the file is too large to read at once." } }, diff --git a/crates/plugin-std/src/check-tests.rs b/crates/plugin-std/src/check-tests.rs index 1426f5c54..cfe83e58a 100644 --- a/crates/plugin-std/src/check-tests.rs +++ b/crates/plugin-std/src/check-tests.rs @@ -4,7 +4,8 @@ use runvil_plugin::ToolErrorKind; use serde_json::json; -use super::check; +use super::{check, invalid_pattern, plural, refuse_blank, refuse_unknown}; +use crate::matcher::PatternError; /// The schema the `StrReplace` tool ships, which has a required string, an /// optional string, and an optional boolean. @@ -138,3 +139,216 @@ fn a_parameter_the_schema_does_not_declare_passes_so_the_tool_can_see_it() { "extra arguments reach the tool, which refuses them itself" ); } + +/// The schema the `Read` tool ships, whose properties are `path`, +/// `offset`, and `limit`. +fn read() -> serde_json::Value { + json!({ + "type": "object", + "properties": { + "path": { "type": "string" }, + "offset": { "type": "integer" }, + "limit": { "type": "integer" } + }, + "required": ["path"] + }) +} + +#[test] +fn an_undeclared_key_is_refused_naming_it_and_listing_the_parameters_sorted() { + let args = json!({ "path": "/a.md", "lines": 3 }); + let Err(error) = refuse_unknown("Read", &read(), &args) else { + panic!("{args} is refused"); + }; + assert_eq!( + error.to_string(), + "std_Read: `lines` is not a parameter of this tool. \ + The parameters are `limit`, `offset` and `path`." + ); + assert_eq!(error.kind(), ToolErrorKind::InvalidArguments); +} + +#[test] +fn the_parameter_list_is_sorted_bytewise_whatever_order_the_schema_holds() { + let schema = json!({ + "type": "object", + "properties": { "zeta": {}, "alpha": {}, "Mid": {} } + }); + let Err(error) = refuse_unknown("Grep", &schema, &json!({ "x": 1 })) else { + panic!("x is refused"); + }; + assert_eq!( + error.to_string(), + "std_Grep: `x` is not a parameter of this tool. \ + The parameters are `Mid`, `alpha` and `zeta`." + ); +} + +#[test] +fn two_parameters_are_joined_with_and_and_one_parameter_has_no_and() { + let two = json!({ "type": "object", "properties": { "b": {}, "a": {} } }); + let Err(error) = refuse_unknown("Read", &two, &json!({ "c": 1 })) else { + panic!("c is refused"); + }; + assert!( + error + .to_string() + .ends_with("The parameters are `a` and `b`."), + "{error}" + ); + let one = json!({ "type": "object", "properties": { "a": {} } }); + let Err(error) = refuse_unknown("Read", &one, &json!({ "c": 1 })) else { + panic!("c is refused"); + }; + assert!( + error.to_string().ends_with("The parameters are `a`."), + "{error}" + ); +} + +#[test] +fn declared_keys_a_null_argument_list_and_a_null_value_pass_the_unknown_key_check() { + assert!(refuse_unknown("Read", &read(), &json!({ "path": "/a.md", "limit": 2 })).is_ok()); + assert!(refuse_unknown("Read", &read(), &json!(null)).is_ok()); + assert!(refuse_unknown("Read", &read(), &json!({ "path": "/a.md", "limit": null })).is_ok()); +} + +#[test] +fn a_null_value_under_an_undeclared_key_is_still_refused() { + let args = json!({ "path": "/a.md", "lines": null }); + assert!(refuse_unknown("Read", &read(), &args).is_err(), "{args}"); +} + +#[test] +fn the_smallest_undeclared_key_is_the_one_named() { + let args = json!({ "path": "/a.md", "zz": 1, "aa": 2, "mm": 3 }); + let Err(error) = refuse_unknown("Read", &read(), &args) else { + panic!("{args} is refused"); + }; + assert!( + error.to_string().contains("`aa` is not a parameter"), + "{error}" + ); +} + +#[test] +fn a_schema_without_properties_refuses_no_key() { + let schema = json!({ "type": "object" }); + assert!(refuse_unknown("Read", &schema, &json!({ "x": 1 })).is_ok()); +} + +#[test] +fn a_schema_with_an_empty_property_list_says_the_tool_takes_none() { + let schema = json!({ "type": "object", "properties": {} }); + let Err(error) = refuse_unknown("Read", &schema, &json!({ "x": 1 })) else { + panic!("x is refused"); + }; + assert_eq!( + error.to_string(), + "std_Read: `x` is not a parameter of this tool. This tool takes no parameters." + ); +} + +#[test] +fn a_blank_value_fails_naming_the_parameter_and_echoing_the_value() { + let Err(error) = refuse_blank("Read", "path", " ") else { + panic!("whitespace is refused"); + }; + assert_eq!( + error.to_string(), + "std_Read: `path` is empty or only whitespace, and the call passed \" \"" + ); + assert_eq!(error.kind(), ToolErrorKind::InvalidArguments); +} + +#[test] +fn an_empty_or_whitespace_only_value_is_blank_and_any_text_is_not() { + for blank in ["", " ", "\t\n"] { + assert!(refuse_blank("Read", "path", blank).is_err(), "{blank:?}"); + } + for text in ["a", " a ", "/notes.md"] { + assert!(refuse_blank("Read", "path", text).is_ok(), "{text:?}"); + } +} + +#[test] +fn plural_adds_s_for_every_count_but_one() { + assert_eq!(plural(1, "line"), "1 line"); + assert_eq!(plural(10, "line"), "10 lines"); + assert_eq!(plural(0, "line"), "0 lines"); + assert_eq!( + plural(2, "invalid UTF-8 sequence"), + "2 invalid UTF-8 sequences" + ); +} + +#[test] +fn plural_turns_a_consonant_then_y_into_ies_and_keeps_a_vowel_then_y() { + assert_eq!(plural(1, "directory"), "1 directory"); + assert_eq!(plural(2, "directory"), "2 directories"); + assert_eq!(plural(0, "entry"), "0 entries"); + assert_eq!(plural(3, "key"), "3 keys"); + assert_eq!(plural(5, "day"), "5 days"); +} + +#[test] +fn a_bad_pattern_names_the_parameter_echoes_the_pattern_and_gives_the_reason() { + let error = invalid_pattern("ListDir", "ignore_globs", "a***", PatternError::StarRun); + assert_eq!( + error.to_string(), + "std_ListDir: `ignore_globs` is invalid: \"a***\" does not parse: \ + more than two consecutive '*' are not supported." + ); + assert_eq!(error.kind(), ToolErrorKind::InvalidArguments); +} + +#[test] +fn a_bad_pattern_gives_each_reason_text_as_the_matcher_words_it() { + for error in [ + PatternError::Empty, + PatternError::TooLong, + PatternError::Control, + PatternError::Backslash, + PatternError::StarRun, + PatternError::PartialDoubleStar, + ] { + let text = invalid_pattern("Grep", "glob", "x", error).to_string(); + assert_eq!( + text, + format!( + "std_Grep: `glob` is invalid: \"x\" does not parse: {}.", + error.reason() + ) + ); + } +} + +#[test] +fn a_pattern_over_80_bytes_is_echoed_cut_with_dots_and_80_bytes_exactly_is_not() { + let eighty = "a".repeat(80); + let kept = invalid_pattern("ListDir", "ignore_globs", &eighty, PatternError::Empty); + assert!( + kept.to_string().contains(&format!("\"{eighty}\" does")), + "{kept}" + ); + let long = "a".repeat(81); + let cut = invalid_pattern("ListDir", "ignore_globs", &long, PatternError::Empty); + assert!( + cut.to_string().contains(&format!("\"{eighty}...\" does")), + "{cut}" + ); +} + +#[test] +fn a_pattern_is_cut_on_a_character_boundary() { + // 79 ASCII bytes and then a 4-byte emoji: byte 80 is inside the emoji, + // so the echo stops at byte 79 rather than splitting it. + let pattern = format!("{}\u{1f4e6}tail", "a".repeat(79)); + let text = + invalid_pattern("ListDir", "ignore_globs", &pattern, PatternError::Empty).to_string(); + assert!( + text.contains(&format!("\"{}...\" does", "a".repeat(79))), + "{text}" + ); + assert!(!text.contains('\u{1f4e6}'), "{text}"); +} diff --git a/crates/plugin-std/src/check.rs b/crates/plugin-std/src/check.rs index c517b7c15..658f9d970 100644 --- a/crates/plugin-std/src/check.rs +++ b/crates/plugin-std/src/check.rs @@ -1,8 +1,12 @@ -//! Argument checking against a tool's own JSON Schema. +//! Argument checking against a tool's own JSON Schema, and the helpers +//! the file tools share to refuse what the schema alone cannot. use runvil_plugin::{ToolError, ToolErrorKind}; use serde_json::Value; +use crate::matcher::PatternError; +use crate::text::clip_to_boundary; + /// Checks that `args` matches the `schema`, which is the `object` schema a /// tool's definition file holds. /// @@ -21,19 +25,22 @@ pub(crate) fn check(tool: &str, schema: &Value, args: &Value) -> Result<(), Tool args }; let Some(object) = args.as_object() else { - return Err(refusal(tool, "the arguments must be a JSON object")); + return Err(invalid_arguments( + tool, + "the arguments must be a JSON object", + )); }; if let Some(required) = schema.get("required").and_then(Value::as_array) { for name in required.iter().filter_map(Value::as_str) { match object.get(name) { None => { - return Err(refusal( + return Err(invalid_arguments( tool, format!("`{name}` is required, and the call left it out"), )); } Some(Value::Null) => { - return Err(refusal( + return Err(invalid_arguments( tool, format!("`{name}` is required, and the call passed null"), )); @@ -60,7 +67,7 @@ fn check_value(tool: &str, name: &str, property: &Value, value: &Value) -> Resul && !allowed.contains(value) { let allowed: Vec = allowed.iter().map(ToString::to_string).collect(); - return Err(refusal( + return Err(invalid_arguments( tool, format!( "`{name}` must be one of {}, and the call passed {value}", @@ -80,7 +87,7 @@ fn check_value(tool: &str, name: &str, property: &Value, value: &Value) -> Resul "object" => value.is_object(), "null" => value.is_null(), other => { - return Err(refusal( + return Err(invalid_arguments( tool, format!("`{name}` has a schema type {other} this build does not know"), )); @@ -89,7 +96,7 @@ fn check_value(tool: &str, name: &str, property: &Value, value: &Value) -> Resul if matches { return check_items(tool, name, property, value); } - Err(refusal( + Err(invalid_arguments( tool, format!("`{name}` must be a JSON {expected}, and the call passed {value}"), )) @@ -110,9 +117,116 @@ fn check_items(tool: &str, name: &str, property: &Value, value: &Value) -> Resul Ok(()) } +/// Refuses a key in `args` that the `schema`'s `properties` do not +/// declare. +/// +/// [`check`] lets an undeclared key through so the tool can decide what to +/// do with it, and the file tools refuse it here, after `check` and before +/// any other work. The message names the smallest undeclared key, and +/// lists the declared parameters sorted bytewise, so the text does not +/// depend on the order the JSON holds them in. A schema with no +/// `properties`, and arguments that are not an object, refuse nothing. +/// +/// # Errors +/// Returns an invalid-arguments [`ToolError`] naming the key. +pub(crate) fn refuse_unknown(tool: &str, schema: &Value, args: &Value) -> Result<(), ToolError> { + let (Some(properties), Some(object)) = ( + schema.get("properties").and_then(Value::as_object), + args.as_object(), + ) else { + return Ok(()); + }; + let Some(unknown) = object + .keys() + .filter(|key| !properties.contains_key(*key)) + .min() + else { + return Ok(()); + }; + let mut names: Vec<&str> = properties.keys().map(String::as_str).collect(); + names.sort_unstable(); + let quoted: Vec = names.iter().map(|name| format!("`{name}`")).collect(); + let tail = match quoted.split_last() { + None => "This tool takes no parameters.".to_owned(), + Some((last, [])) => format!("The parameters are {last}."), + Some((last, first)) => format!("The parameters are {} and {last}.", first.join(", ")), + }; + Err(invalid_arguments( + tool, + format!("`{unknown}` is not a parameter of this tool. {tail}"), + )) +} + +/// Refuses `value` when it is empty or only whitespace, naming the +/// parameter `name` and echoing the value. +/// +/// This runs before any I/O, so a blank path never reaches the +/// filesystem. +/// +/// # Errors +/// Returns an invalid-arguments [`ToolError`] naming the parameter. +pub(crate) fn refuse_blank(tool: &str, name: &str, value: &str) -> Result<(), ToolError> { + if value.trim().is_empty() { + return Err(invalid_arguments( + tool, + format!("`{name}` is empty or only whitespace, and the call passed \"{value}\""), + )); + } + Ok(()) +} + +/// The most bytes of a refused pattern the error echoes. +const PATTERN_ECHO_BYTES: usize = 80; + +/// The invalid-arguments error for the pattern `pattern`, which the +/// matcher refused with `error`, passed under the parameter `name`. +/// +/// The error echoes the pattern, cut to 80 bytes on a character boundary +/// with `...` added when it was cut, so a long pattern does not fill the +/// message. +pub(crate) fn invalid_pattern( + tool: &str, + name: &str, + pattern: &str, + error: PatternError, +) -> ToolError { + let shown = clip_to_boundary(pattern, 0, PATTERN_ECHO_BYTES); + let dots = if shown.len() < pattern.len() { + "..." + } else { + "" + }; + invalid_arguments( + tool, + format!( + "`{name}` is invalid: \"{shown}{dots}\" does not parse: {}.", + error.reason() + ), + ) +} + +/// `{n} {noun}` for a count, with `s` added to the noun for any `n` but 1. +/// +/// A noun that ends in a consonant and then `y` takes `ies` instead, so +/// `directory` gives `directories`. Examples: `1 line`, `10 lines`, +/// `0 lines`. +pub(crate) fn plural(n: usize, noun: &str) -> String { + if n == 1 { + return format!("{n} {noun}"); + } + if let Some(stem) = noun.strip_suffix('y') + && let Some(before) = stem.chars().next_back() + && before.is_alphabetic() + && !matches!(before.to_ascii_lowercase(), 'a' | 'e' | 'i' | 'o' | 'u') + { + return format!("{n} {stem}ies"); + } + format!("{n} {noun}s") +} + /// An invalid-arguments error whose message opens with the tool's id /// segment. -fn refusal(tool: &str, reason: impl std::fmt::Display) -> ToolError { +pub(crate) fn invalid_arguments(tool: &str, reason: impl std::fmt::Display) -> ToolError { ToolError::new( ToolErrorKind::InvalidArguments, format!("std_{tool}: {reason}"), diff --git a/crates/plugin-std/src/dir.rs b/crates/plugin-std/src/dir.rs new file mode 100644 index 000000000..6b20d4709 --- /dev/null +++ b/crates/plugin-std/src/dir.rs @@ -0,0 +1,72 @@ +//! The `ListDir` tool: a bullet tree of a directory, listed breadth first. +//! +//! ListDir stats the directory the call names and walks it through the +//! call's [`Access`], so the run's mode and policy apply. The tree holds +//! at most 15,000 bytes of lines: when the next line would pass that, the +//! walk stops and the answer says so. Directories end in `/`, files and +//! directories share one byte-ordered list, entries named `.git` are +//! skipped, and a link is listed and never entered. The names are +//! untrusted output. + +mod args; +mod render; +mod walk; + +#[cfg(test)] +mod tests; + +use runvil_plugin::{Access, FileType, ToolError, ToolOutput, VfsError}; +use serde_json::Value; + +use crate::refusal::{missing_root, not_a_directory, refusal}; +use crate::spec::Definition; +use crate::text::BODY_CAP; + +use self::args::Request; + +/// Runs one `ListDir` call whose arguments already passed the shared +/// schema check. +/// +/// # Errors +/// Returns an invalid-arguments [`ToolError`] naming the parameter for a +/// bad argument, and a message error for any other failure, worded for a +/// model to read. +pub(crate) fn call( + access: &Access, + definition: &Definition, + args: &Value, +) -> Result { + let request = args::parse(definition, args)?; + list_directory(access, &definition.wire_name, &request, BODY_CAP) +} + +/// Lists the directory `request` names, with at most `cap` bytes of tree +/// lines. `tool` is the name the model called the tool by. +fn list_directory( + access: &Access, + tool: &str, + request: &Request<'_>, + cap: usize, +) -> Result { + let target = request.target; + let failed = |error: VfsError| ToolError::message(refusal(tool, target, &error)); + let stat = match access.stat(target) { + Ok(stat) => stat, + Err(VfsError::NotFound { .. }) => return Err(ToolError::message(missing_root(target))), + Err(error) => return Err(failed(error)), + }; + check_directory(target, stat.file_type).map_err(ToolError::message)?; + let tree = walk::walk(target, &request.ignore, cap, |path| access.list(path)) + .map_err(|failure| ToolError::message(refusal(tool, &failure.path, &failure.error)))?; + Ok(ToolOutput::untrusted(render::render(&tree))) +} + +/// Decides from the kind of node at `path` whether ListDir goes on: a +/// directory does, and so does a link, which is walked through, and +/// anything else gets the sentence that says it is not a directory. +fn check_directory(path: &str, file_type: FileType) -> Result<(), String> { + match file_type { + FileType::Directory | FileType::Symlink => Ok(()), + _ => Err(not_a_directory(path)), + } +} diff --git a/crates/plugin-std/src/dir/args.rs b/crates/plugin-std/src/dir/args.rs new file mode 100644 index 000000000..52abc9ef9 --- /dev/null +++ b/crates/plugin-std/src/dir/args.rs @@ -0,0 +1,52 @@ +//! ListDir's arguments: what the call asks for, and what is refused before +//! any path is looked at. + +use runvil_plugin::ToolError; +use serde_json::Value; + +use crate::check::{invalid_pattern, refuse_blank, refuse_unknown}; +use crate::matcher::Matcher; +use crate::spec::Definition; + +/// What a call asks ListDir for. +#[derive(Debug)] +pub(super) struct Request<'a> { + /// The directory to list, as the model wrote it. + pub(super) target: &'a str, + /// The `ignore_globs`, compiled, in the order the call gave them. + pub(super) ignore: Vec, +} + +/// Reads the call's arguments after the shared schema check. +/// +/// The order is fixed: an undeclared key, then a blank target, then each +/// `ignore_globs` pattern in turn, so a call that breaks two rules always +/// hears about the same one. +/// +/// # Errors +/// Returns an invalid-arguments [`ToolError`] naming the parameter. A +/// refused pattern is echoed with the rule it broke. +pub(super) fn parse<'a>( + definition: &Definition, + args: &'a Value, +) -> Result, ToolError> { + let tool = definition.wire_name.as_str(); + refuse_unknown(tool, &definition.parameters, args)?; + let target = args + .get("target_directory") + .and_then(Value::as_str) + .unwrap_or_default(); + refuse_blank(tool, "target_directory", target)?; + let ignore = args + .get("ignore_globs") + .and_then(Value::as_array) + .map(|patterns| patterns.iter().filter_map(Value::as_str)) + .into_iter() + .flatten() + .map(|pattern| { + Matcher::new(pattern) + .map_err(|error| invalid_pattern(tool, "ignore_globs", pattern, error)) + }) + .collect::, ToolError>>()?; + Ok(Request { target, ignore }) +} diff --git a/crates/plugin-std/src/dir/render.rs b/crates/plugin-std/src/dir/render.rs new file mode 100644 index 000000000..6a00b073e --- /dev/null +++ b/crates/plugin-std/src/dir/render.rs @@ -0,0 +1,73 @@ +//! The render: turns the walk's tree into the text ListDir answers with. +//! +//! The tree prints depth first in the sorted order, one line per entry, +//! each indented two spaces per level under a `- ` bullet, with a trailing +//! `/` on a directory only. The lines are joined with `\n` and there is no +//! newline after the last one. + +use super::walk::{Node, Tree}; + +/// The note a cut listing ends with, after the ` ...` line and a blank +/// line. +const TOO_LARGE: &str = "target_directory is too large (either too many children or their \ + names are too long). Immediate children were truncated. Try a more specific \ + target_directory or ignore_globs."; + +/// The line that follows the root line when nothing is left to show. +const NO_CHILDREN: &str = "... no children found ..."; + +/// The line that marks where a cut listing stops. +const CUT_MARK: &str = " ..."; + +/// The text of `tree`. +/// +/// A root with nothing to show prints its line and then `... no children +/// found ...`. A tree the walk cut ends with the cut mark, a blank line, +/// and the note that says to narrow the call. The cut mark and the note +/// do not count against the byte cap. +pub(super) fn render(tree: &Tree) -> String { + // The walk always seeds the root as the first node and only pushes + // indexes it has added, so indexing the nodes cannot miss. A broken + // tree panics here instead of printing less than the walk found. + let root = &tree.nodes[0]; + if root.children.is_empty() && !tree.truncated { + return format!("{}\n{NO_CHILDREN}", tree.root_line); + } + let mut lines = vec![tree.root_line.clone()]; + let mut stack: Vec<(usize, usize)> = root + .children + .iter() + .rev() + .map(|child| (*child, 1)) + .collect(); + while let Some((index, depth)) = stack.pop() { + let node = &tree.nodes[index]; + lines.push(line(node, depth)); + stack.extend(node.children.iter().rev().map(|child| (*child, depth + 1))); + } + let mut text = lines.join("\n"); + if tree.truncated { + text.push('\n'); + text.push_str(CUT_MARK); + text.push_str("\n\n"); + text.push_str(TOO_LARGE); + } + text +} + +/// The line of `node` at `depth`: the indent, the bullet, the name, a `/` +/// for a directory, and `...` for a directory that was shown but never +/// expanded. +fn line(node: &Node, depth: usize) -> String { + let slash = if node.directory { "/" } else { "" }; + let unexpanded = if node.directory && !node.expanded { + "..." + } else { + "" + }; + format!( + "{}- {}{slash}{unexpanded}", + " ".repeat(2 * depth), + node.name + ) +} diff --git a/crates/plugin-std/src/dir/tests.rs b/crates/plugin-std/src/dir/tests.rs new file mode 100644 index 000000000..83c6aa5f1 --- /dev/null +++ b/crates/plugin-std/src/dir/tests.rs @@ -0,0 +1,84 @@ +//! Unit tests for the `ListDir` tool. This file holds the helpers the +//! topic files share: a call over a fresh in-memory filesystem, a seeded +//! tree, and the two ways to drive ListDir, through its arguments and +//! through a lowered byte cap. + +mod arguments; +mod cap; +mod ignore; +mod tree; + +use runvil_plugin::testing::TestCall; +use runvil_plugin::{OutputTrust, ToolError, ToolId, ToolOutput}; +use serde_json::Value; + +use super::args::Request; +use crate::matcher::Matcher; +use crate::spec::{self, Definition}; + +/// A call of `std/dir` over a fresh in-memory filesystem. +fn new_call() -> TestCall { + TestCall::new(ToolId::parse("std/dir").expect("a two-segment tool id parses")) +} + +/// A call with one small file at each of `files`, which makes every +/// directory on the way to it. +fn seeded(files: &[&str]) -> TestCall { + let call = new_call(); + for path in files { + call.context() + .access() + .write(path, b"x") + .expect("a seeded file is written"); + } + call +} + +/// The `dir` definition the Pack builds the tool from. +fn definition() -> &'static Definition { + spec::all() + .expect("every embedded definition parses") + .iter() + .find(|definition| definition.segment == "dir") + .expect("the dir definition exists") +} + +/// Calls ListDir with `args` as the Pack does: the schema check first, +/// then the tool. +fn list(call: &TestCall, args: &Value) -> Result { + let definition = definition(); + crate::check::check(&definition.wire_name, &definition.parameters, args)?; + super::call(call.context().access(), definition, args) +} + +/// Lists `target`, leaving out `ignore`, under the byte cap `cap`. +fn list_capped( + call: &TestCall, + target: &str, + ignore: &[&str], + cap: usize, +) -> Result { + let request = Request { + target, + ignore: ignore + .iter() + .map(|pattern| Matcher::new(pattern).expect("the pattern is valid")) + .collect(), + }; + super::list_directory(call.context().access(), "ListDir", &request, cap) +} + +/// The text of a successful listing, which must be untrusted output. +fn text_of(result: Result) -> String { + let output = result.expect("the listing succeeds"); + assert_eq!(output.trust(), OutputTrust::Untrusted, "{}", output.text()); + output.text().to_owned() +} + +/// The message of a failed listing. +fn failure_of(result: Result) -> String { + match result { + Ok(output) => panic!("the listing fails, but it answered {:?}", output.text()), + Err(error) => error.to_string(), + } +} diff --git a/crates/plugin-std/src/dir/tests/arguments.rs b/crates/plugin-std/src/dir/tests/arguments.rs new file mode 100644 index 000000000..4c62b90e7 --- /dev/null +++ b/crates/plugin-std/src/dir/tests/arguments.rs @@ -0,0 +1,80 @@ +//! What ListDir refuses about its arguments, and in which order. + +use runvil_plugin::ToolErrorKind; +use serde_json::json; + +use super::{failure_of, list, new_call}; + +#[test] +fn a_missing_target_names_the_parameter() { + let call = new_call(); + let error = list(&call, &json!({})).expect_err("target_directory is required"); + assert_eq!( + error.to_string(), + "std_ListDir: `target_directory` is required, and the call left it out" + ); + assert_eq!(error.kind(), ToolErrorKind::InvalidArguments); +} + +#[test] +fn a_target_that_is_not_a_string_names_the_parameter() { + let call = new_call(); + assert_eq!( + failure_of(list(&call, &json!({ "target_directory": 7 }))), + "std_ListDir: `target_directory` must be a JSON string, and the call passed 7" + ); +} + +#[test] +fn ignore_globs_must_be_an_array_of_strings() { + let call = new_call(); + assert_eq!( + failure_of(list( + &call, + &json!({ "target_directory": "/", "ignore_globs": "*.log" }) + )), + "std_ListDir: `ignore_globs` must be a JSON array, and the call passed \"*.log\"" + ); + assert_eq!( + failure_of(list( + &call, + &json!({ "target_directory": "/", "ignore_globs": ["a", 3] }) + )), + "std_ListDir: `ignore_globs` must be a JSON string, and the call passed 3" + ); +} + +#[test] +fn an_undeclared_key_is_refused_and_the_parameters_are_named() { + let call = new_call(); + let error = list(&call, &json!({ "target_directory": "/", "path": "/x" })) + .expect_err("path is not a parameter"); + assert_eq!( + error.to_string(), + "std_ListDir: `path` is not a parameter of this tool. \ + The parameters are `ignore_globs` and `target_directory`." + ); + assert_eq!(error.kind(), ToolErrorKind::InvalidArguments); +} + +#[test] +fn an_undeclared_key_is_refused_before_a_blank_target() { + let call = new_call(); + assert_eq!( + failure_of(list(&call, &json!({ "target_directory": " ", "depth": 2 }))), + "std_ListDir: `depth` is not a parameter of this tool. \ + The parameters are `ignore_globs` and `target_directory`." + ); +} + +#[test] +fn a_blank_target_is_refused_before_a_bad_pattern() { + let call = new_call(); + assert_eq!( + failure_of(list( + &call, + &json!({ "target_directory": " ", "ignore_globs": ["a***"] }) + )), + "std_ListDir: `target_directory` is empty or only whitespace, and the call passed \" \"" + ); +} diff --git a/crates/plugin-std/src/dir/tests/cap.rs b/crates/plugin-std/src/dir/tests/cap.rs new file mode 100644 index 000000000..a3eaa529d --- /dev/null +++ b/crates/plugin-std/src/dir/tests/cap.rs @@ -0,0 +1,181 @@ +//! The byte cap: the walk is breadth first, stops when the next line would +//! pass the cap, marks what it did not expand, and ends with its note. + +use std::collections::HashSet; + +use runvil_plugin::OutputTrust; +use runvil_plugin::testing::TestCall; + +use super::{list_capped, seeded, text_of}; +use crate::text::BODY_CAP; + +/// The note the output ends with after the ` ...` line and a blank +/// line. +const TOO_LARGE: &str = "target_directory is too large (either too many children or their names \ + are too long). Immediate children were truncated. Try a more specific target_directory or \ + ignore_globs."; + +/// Two files at the top and two small directories, each with one file. +/// +/// The lines, with the bytes each counts with its newline, are the root +/// `/p/` (4), ` - a.txt` (10), ` - b/` (7), ` - c/` (7), ` - x.txt` +/// (12), and ` - y.txt` (12). +fn small() -> TestCall { + seeded(&["/p/a.txt", "/p/b/x.txt", "/p/c/y.txt"]) +} + +#[test] +fn the_default_cap_is_fifteen_thousand_bytes() { + assert_eq!(BODY_CAP, 15_000); +} + +#[test] +fn a_tree_that_fits_has_no_note() { + let call = small(); + let text = text_of(list_capped(&call, "/p", &[], 52)); + assert_eq!( + text, + "/p/\n - a.txt\n - b/\n - x.txt\n - c/\n - y.txt" + ); +} + +#[test] +fn a_tree_that_just_passes_the_cap_stops_at_the_last_line_that_fits() { + let call = small(); + let text = text_of(list_capped(&call, "/p", &[], 51)); + let expected = + format!("/p/\n - a.txt\n - b/\n - x.txt\n - c/...\n ...\n\n{TOO_LARGE}"); + assert_eq!(text, expected); +} + +#[test] +fn the_note_follows_a_dots_line_and_a_blank_line_byte_for_byte() { + let call = small(); + let text = text_of(list_capped(&call, "/p", &[], 40)); + let expected = [ + "/p/", + " - a.txt", + " - b/", + " - x.txt", + " - c/...", + " ...", + "", + TOO_LARGE, + ] + .join("\n"); + assert_eq!(text, expected); + assert!(text.ends_with("ignore_globs.")); +} + +#[test] +fn a_directory_whose_first_child_does_not_fit_is_marked_unexpanded() { + let call = small(); + let text = text_of(list_capped(&call, "/p", &[], 39)); + let expected = format!("/p/\n - a.txt\n - b/...\n - c/...\n ...\n\n{TOO_LARGE}"); + assert_eq!(text, expected); +} + +#[test] +fn the_directory_being_listed_keeps_the_children_already_added() { + // Cap 21 holds the root, `a.txt`, and `b/`. The root's `c/` does not + // fit, so the walk stops with `c` not in the tree at all, and `b` is + // shown but never expanded. + let call = small(); + let text = text_of(list_capped(&call, "/p", &[], 21)); + let expected = format!("/p/\n - a.txt\n - b/...\n ...\n\n{TOO_LARGE}"); + assert_eq!(text, expected); +} + +#[test] +fn the_walk_reaches_the_top_levels_before_going_deeper() { + // Breadth first: with room for the root's children and one more line, + // the one more line is the first child of the first directory, not a + // grandchild reached by going deep first. + let call = seeded(&["/r/a/deep/f.txt", "/r/a/g.txt", "/r/b/h.txt", "/r/z.txt"]); + let text = text_of(list_capped(&call, "/r", &[], 40)); + assert!(text.contains(" - a/\n - deep/...\n"), "{text}"); + assert!(text.contains(" - b/...\n"), "{text}"); + assert!(!text.contains("f.txt"), "{text}"); +} + +#[test] +fn nothing_fits_when_the_cap_is_smaller_than_the_root_line() { + let call = small(); + let text = text_of(list_capped(&call, "/p", &[], 3)); + assert_eq!(text, format!("/p/\n ...\n\n{TOO_LARGE}")); +} + +#[test] +fn the_note_and_the_dots_do_not_count_against_the_cap() { + // The lines that stay in the tree use at most the cap, though the + // output is longer because of the note and the `...` suffixes. + let call = small(); + let text = text_of(list_capped(&call, "/p", &[], 40)); + let kept: usize = text + .lines() + .take_while(|line| *line != " ...") + .map(|line| line.trim_end_matches("...").len() + 1) + .sum(); + assert!(kept <= 40, "{kept} bytes of tree lines in {text}"); + assert!(text.len() > 40); +} + +#[test] +fn a_truncated_listing_is_a_successful_untrusted_answer() { + let call = small(); + let output = list_capped(&call, "/p", &[], 21).expect("a truncated listing succeeds"); + assert_eq!(output.trust(), OutputTrust::Untrusted); +} + +#[test] +fn a_cap_sweep_over_multi_byte_names_never_cuts_inside_a_name_or_a_line() { + let files = [ + "/w/é/日本語/ファイル.txt", + "/w/é/ü.md", + "/w/📦pkg/readme.md", + "/w/📦pkg/📄.txt", + "/w/日本/a.md", + "/w/ñandú.rs", + "/w/z.txt", + ]; + let call = seeded(&files); + let full = text_of(list_capped(&call, "/w", &[], BODY_CAP)); + assert!(!full.contains("target_directory is too large"), "{full}"); + let names: HashSet<&str> = full + .lines() + .skip(1) + .map(|line| { + line.trim_start() + .trim_start_matches("- ") + .trim_end_matches('/') + }) + .collect(); + let all_bytes = full.len() + 1; + for cap in 0..=all_bytes { + let text = text_of(list_capped(&call, "/w", &[], cap)); + let mut lines = text.lines(); + assert_eq!(lines.next(), Some("/w/"), "cap {cap}"); + let mut kept = "/w/".len() + 1; + let mut seen_note = false; + for line in lines { + if line == " ..." { + seen_note = true; + break; + } + let stripped = line.trim_end_matches("..."); + assert!(line.starts_with(' '), "cap {cap}: {line:?}"); + let name = stripped + .trim_start() + .strip_prefix("- ") + .unwrap_or_else(|| panic!("cap {cap}: {line:?} is not a bullet line")) + .trim_end_matches('/'); + assert!(names.contains(name), "cap {cap}: {name:?} is cut in {text}"); + kept += stripped.len() + 1; + } + assert_eq!(seen_note, cap < all_bytes, "cap {cap}: {text}"); + if seen_note { + assert!(text.ends_with(TOO_LARGE), "cap {cap}: {text}"); + assert!(kept <= cap.max(4), "cap {cap}: {kept} bytes kept in {text}"); + } + } +} diff --git a/crates/plugin-std/src/dir/tests/ignore.rs b/crates/plugin-std/src/dir/tests/ignore.rs new file mode 100644 index 000000000..0ca4bee92 --- /dev/null +++ b/crates/plugin-std/src/dir/tests/ignore.rs @@ -0,0 +1,185 @@ +//! `ignore_globs`: where a pattern matches, what a match leaves out, and +//! what a refused pattern says. + +use runvil_plugin::{Entry, FileType, Stat, ToolErrorKind, VfsError}; +use serde_json::json; + +use super::{failure_of, list, seeded, text_of}; +use crate::dir::walk::walk; +use crate::matcher::Matcher; + +/// A tree with `node_modules`, `.log` files, and a `build` directory at +/// the root and one level down. +const FILES: [&str; 7] = [ + "/node_modules/x.js", + "/src/node_modules/y.js", + "/src/a.log", + "/b.log", + "/build/out.o", + "/src/build/z.o", + "/src/app.rs", +]; + +#[test] +fn a_pattern_with_no_leading_slash_leaves_out_entries_at_every_level() { + let call = seeded(&FILES); + let args = json!({ "target_directory": "/", "ignore_globs": ["node_modules", "*.log"] }); + let expected = "\ +/ + - build/ + - out.o + - src/ + - app.rs + - build/ + - z.o"; + assert_eq!(text_of(list(&call, &args)), expected); +} + +#[test] +fn a_pattern_with_a_leading_slash_leaves_out_only_the_entry_at_that_path() { + let call = seeded(&FILES); + let args = json!({ "target_directory": "/", "ignore_globs": ["/build"] }); + let expected = "\ +/ + - b.log + - node_modules/ + - x.js + - src/ + - a.log + - app.rs + - build/ + - z.o + - node_modules/ + - y.js"; + assert_eq!(text_of(list(&call, &args)), expected); +} + +#[test] +fn a_pattern_is_matched_against_the_path_below_the_target_as_written() { + let call = seeded(&FILES); + let args = json!({ "target_directory": "/src", "ignore_globs": ["/src/build", "a.log"] }); + assert_eq!( + text_of(list(&call, &args)), + "/src/\n - app.rs\n - node_modules/\n - y.js" + ); +} + +#[test] +fn a_pattern_can_name_a_path_below_a_relative_target() { + // The path a pattern is matched against has a leading `/` even when + // the target was written without one. + let call = seeded(&FILES); + let args = json!({ "target_directory": "src", "ignore_globs": ["/src/build"] }); + assert_eq!( + text_of(list(&call, &args)), + "/src/\n - a.log\n - app.rs\n - node_modules/\n - y.js" + ); +} + +#[test] +fn a_double_star_pattern_leaves_out_everything_below() { + let call = seeded(&FILES); + let args = json!({ "target_directory": "/", "ignore_globs": ["src/**"] }); + assert_eq!( + text_of(list(&call, &args)), + "/\n - b.log\n - build/\n - out.o\n - node_modules/\n - x.js\n - src/" + ); +} + +#[test] +fn a_matching_directory_is_not_entered() { + let lister = |path: &str| -> Result, VfsError> { + let directory = |name: &str| Entry::new(name, Stat::new(FileType::Directory, 0)); + match path { + "/p" => Ok(vec![directory("node_modules"), directory("src")]), + "/p/src" => Ok(vec![Entry::new("a.rs", Stat::new(FileType::File, 0))]), + other => panic!("the walk listed {other}"), + } + }; + let ignore = [Matcher::new("node_modules").expect("the pattern is valid")]; + let tree = walk("/p", &ignore, 15_000, lister).expect("the walk succeeds"); + assert_eq!( + crate::dir::render::render(&tree), + "/p/\n - src/\n - a.rs" + ); +} + +#[test] +fn a_pattern_that_matches_nothing_changes_nothing() { + let call = seeded(&["/p/a.md", "/p/sub/b.md"]); + let with = text_of(list( + &call, + &json!({ "target_directory": "/p", "ignore_globs": ["nothing-here"] }), + )); + let without = text_of(list(&call, &json!({ "target_directory": "/p" }))); + assert_eq!(with, without); +} + +#[test] +fn an_empty_list_of_patterns_and_a_null_one_leave_nothing_out() { + let call = seeded(&["/p/a.md"]); + let plain = text_of(list(&call, &json!({ "target_directory": "/p" }))); + for ignore in [json!([]), json!(null)] { + let args = json!({ "target_directory": "/p", "ignore_globs": ignore }); + assert_eq!(text_of(list(&call, &args)), plain, "{args}"); + } +} + +#[test] +fn a_refused_pattern_is_an_invalid_arguments_error_with_the_reason() { + let call = seeded(&["/p/a.md"]); + for (pattern, reason) in [ + ("a***", "more than two consecutive '*' are not supported"), + ("a/**b", "'**' must occupy a whole path segment"), + ("", "pattern is empty"), + ("a\\b", "pattern does not support backslash escapes"), + ("a\u{1}b", "pattern contains a control character"), + ] { + let args = json!({ "target_directory": "/p", "ignore_globs": [pattern] }); + let error = list(&call, &args).expect_err("the pattern is refused"); + assert_eq!(error.kind(), ToolErrorKind::InvalidArguments, "{pattern:?}"); + assert_eq!( + error.to_string(), + format!( + "std_ListDir: `ignore_globs` is invalid: \"{pattern}\" does not parse: {reason}." + ) + ); + } +} + +#[test] +fn an_over_long_pattern_is_echoed_cut_to_80_bytes_with_dots() { + let call = seeded(&["/p/a.md"]); + let pattern = format!("{}***", "a".repeat(100)); + let args = json!({ "target_directory": "/p", "ignore_globs": [pattern] }); + assert_eq!( + failure_of(list(&call, &args)), + format!( + "std_ListDir: `ignore_globs` is invalid: \"{}...\" does not parse: \ + more than two consecutive '*' are not supported.", + "a".repeat(80) + ) + ); +} + +#[test] +fn the_first_refused_pattern_is_the_one_named() { + let call = seeded(&["/p/a.md"]); + let args = json!({ "target_directory": "/p", "ignore_globs": ["ok", "a***", "a/**b"] }); + assert_eq!( + failure_of(list(&call, &args)), + "std_ListDir: `ignore_globs` is invalid: \"a***\" does not parse: \ + more than two consecutive '*' are not supported." + ); +} + +#[test] +fn a_pattern_is_refused_before_the_path_is_looked_at() { + let call = seeded(&[]); + let args = json!({ "target_directory": "/does/not/exist", "ignore_globs": ["a***"] }); + assert_eq!( + failure_of(list(&call, &args)), + "std_ListDir: `ignore_globs` is invalid: \"a***\" does not parse: \ + more than two consecutive '*' are not supported." + ); +} diff --git a/crates/plugin-std/src/dir/tests/tree.rs b/crates/plugin-std/src/dir/tests/tree.rs new file mode 100644 index 000000000..82f06b228 --- /dev/null +++ b/crates/plugin-std/src/dir/tests/tree.rs @@ -0,0 +1,458 @@ +//! The shape of the tree ListDir answers with: the layout, the order, what +//! is skipped, how a link is listed, and what each failure says. + +use runvil_plugin::{Entry, FileType, OutputTrust, Stat, VfsError}; +use serde_json::json; + +use super::{failure_of, list, new_call, seeded, text_of}; +use crate::dir::check_directory; +use crate::dir::render::render; +use crate::dir::walk::{Failure, walk}; +use crate::refusal::refusal; + +/// The listing ListDir answers for `target`. +fn tree(files: &[&str], target: &str) -> String { + text_of(list(&seeded(files), &json!({ "target_directory": target }))) +} + +/// A directory entry of kind `file_type`. +fn entry(name: &str, file_type: FileType) -> Entry { + Entry::new(name, Stat::new(file_type, 0)) +} + +#[test] +fn the_posix_tree_lists_every_level_in_byte_order() { + // The parent report shows `package.json` before `README.md`, because + // Cursor sorts with `localeCompare`. Byte order governs here, so + // `README.md` comes first. + let text = tree( + &[ + "/home/dev/proj/README.md", + "/home/dev/proj/package.json", + "/home/dev/proj/docs/guide.md", + "/home/dev/proj/src/app.tsx", + "/home/dev/proj/src/index.ts", + "/home/dev/proj/src/utils/math.ts", + ], + "/home/dev/proj", + ); + let expected = "\ +/home/dev/proj/ + - README.md + - docs/ + - guide.md + - package.json + - src/ + - app.tsx + - index.ts + - utils/ + - math.ts"; + assert_eq!(text, expected); +} + +#[test] +fn the_output_has_no_trailing_newline() { + let text = tree(&["/p/a.md"], "/p"); + assert_eq!(text, "/p/\n - a.md"); + assert!(!text.ends_with('\n')); +} + +#[test] +fn an_empty_directory_says_it_has_no_children() { + let call = new_call(); + call.context() + .access() + .mkdir("/home/dev/proj/empty", true) + .expect("the directory is made"); + assert_eq!( + text_of(list( + &call, + &json!({ "target_directory": "/home/dev/proj/empty" }) + )), + "/home/dev/proj/empty/\n... no children found ..." + ); +} + +#[test] +fn a_root_holding_only_dot_git_says_it_has_no_children() { + let call = seeded(&["/repo/.git/HEAD", "/repo/.git/objects/ab"]); + assert_eq!( + text_of(list(&call, &json!({ "target_directory": "/repo" }))), + "/repo/\n... no children found ..." + ); +} + +#[test] +fn a_root_whose_children_are_all_ignored_says_it_has_no_children() { + let call = seeded(&["/p/a.log", "/p/b.log"]); + assert_eq!( + text_of(list( + &call, + &json!({ "target_directory": "/p", "ignore_globs": ["*.log"] }) + )), + "/p/\n... no children found ..." + ); +} + +#[test] +fn a_directory_that_is_empty_below_the_root_is_still_listed() { + let call = seeded(&["/p/a.md"]); + call.context() + .access() + .mkdir("/p/empty", false) + .expect("the directory is made"); + assert_eq!( + text_of(list(&call, &json!({ "target_directory": "/p" }))), + "/p/\n - a.md\n - empty/" + ); +} + +#[test] +fn a_file_as_the_target_is_not_a_directory() { + let call = seeded(&["/home/dev/proj/README.md"]); + assert_eq!( + failure_of(list( + &call, + &json!({ "target_directory": "/home/dev/proj/README.md" }) + )), + "Path is not a directory: /home/dev/proj/README.md" + ); +} + +#[test] +fn a_missing_path_does_not_exist() { + let call = new_call(); + assert_eq!( + failure_of(list(&call, &json!({ "target_directory": "/nope" }))), + "Path does not exist: /nope" + ); +} + +#[test] +fn a_failure_echoes_the_path_as_the_model_wrote_it() { + let call = seeded(&["/notes/a.md"]); + assert_eq!( + failure_of(list(&call, &json!({ "target_directory": "nope" }))), + "Path does not exist: nope" + ); + assert_eq!( + failure_of(list(&call, &json!({ "target_directory": "notes/a.md" }))), + "Path is not a directory: notes/a.md" + ); +} + +#[test] +fn a_blank_target_is_refused_before_any_io() { + let call = new_call(); + for blank in ["", " ", "\t\n"] { + assert_eq!( + failure_of(list(&call, &json!({ "target_directory": blank }))), + format!( + "std_ListDir: `target_directory` is empty or only whitespace, \ + and the call passed \"{blank}\"" + ) + ); + } +} + +#[test] +fn the_file_type_decision_covers_every_kind_of_node() { + assert_eq!(check_directory("/p", FileType::Directory), Ok(())); + assert_eq!(check_directory("/p", FileType::Symlink), Ok(())); + for other in [ + FileType::File, + FileType::Fifo, + FileType::Socket, + FileType::CharDevice, + FileType::BlockDevice, + ] { + assert_eq!( + check_directory("/p", other), + Err("Path is not a directory: /p".to_owned()), + "{other:?}" + ); + } +} + +#[test] +fn a_relative_target_gets_a_leading_slash_on_the_root_line() { + let call = seeded(&["/home/dev/a.md"]); + assert_eq!( + text_of(list(&call, &json!({ "target_directory": "home/dev" }))), + "/home/dev/\n - a.md" + ); +} + +#[test] +fn a_trailing_slash_on_the_target_is_not_doubled() { + let call = seeded(&["/home/dev/a.md"]); + assert_eq!( + text_of(list(&call, &json!({ "target_directory": "/home/dev/" }))), + "/home/dev/\n - a.md" + ); +} + +#[test] +fn the_filesystem_root_is_a_single_slash() { + let call = seeded(&["/a.md", "/b/c.md"]); + assert_eq!( + text_of(list(&call, &json!({ "target_directory": "/" }))), + "/\n - a.md\n - b/\n - c.md" + ); +} + +#[test] +fn dot_git_is_skipped_while_other_dotfiles_are_listed() { + let call = seeded(&[ + "/p/.git/HEAD", + "/p/.env", + "/p/.gitignore", + "/p/src/.hidden", + "/p/src/.git/config", + "/p/src/a.rs", + ]); + let expected = "\ +/p/ + - .env + - .gitignore + - src/ + - .hidden + - a.rs"; + assert_eq!( + text_of(list(&call, &json!({ "target_directory": "/p" }))), + expected + ); +} + +#[test] +fn a_dot_git_that_is_a_file_is_skipped_too() { + let call = seeded(&["/p/.git", "/p/a.md", "/p/sub/.git"]); + assert_eq!( + text_of(list(&call, &json!({ "target_directory": "/p" }))), + "/p/\n - a.md\n - sub/" + ); +} + +#[test] +fn a_name_that_only_starts_with_dot_git_is_listed() { + let call = seeded(&["/p/.github/ci.yml", "/p/.gitmodules"]); + assert_eq!( + text_of(list(&call, &json!({ "target_directory": "/p" }))), + "/p/\n - .github/\n - ci.yml\n - .gitmodules" + ); +} + +#[test] +fn the_sort_order_is_the_order_access_glob_gives() { + let call = seeded(&["/B", "/a-b/x", "/a.txt", "/a/x", "/b"]); + assert_eq!( + text_of(list(&call, &json!({ "target_directory": "/" }))), + "/\n - B\n - a-b/\n - x\n - a.txt\n - a/\n - x\n - b" + ); + let globbed = call + .context() + .access() + .glob("/**") + .expect("the pattern is valid"); + assert_eq!(globbed, ["/B", "/a-b/x", "/a.txt", "/a/x", "/b"]); +} + +#[test] +fn a_directory_sorts_by_its_name_with_a_slash_added() { + // `a` the directory sorts as `a/`, which is after `a-b` and `a.txt`, + // because `/` is 0x2f and `-` and `.` are lower. + let call = seeded(&["/a/x", "/a-b", "/a.txt", "/a0"]); + let text = text_of(list(&call, &json!({ "target_directory": "/" }))); + let names: Vec<&str> = text + .lines() + .skip(1) + .filter(|line| line.starts_with(" - ")) + .collect(); + assert_eq!(names, [" - a-b", " - a.txt", " - a/", " - a0"]); +} + +#[test] +fn a_symlink_is_listed_by_name_without_a_slash_and_is_never_entered() { + let lister = |path: &str| -> Result, VfsError> { + match path { + "/p" => Ok(vec![ + entry("link", FileType::Symlink), + entry("sub", FileType::Directory), + entry("file", FileType::File), + ]), + "/p/sub" => Ok(vec![entry("inner", FileType::File)]), + other => panic!("the walk entered {other}"), + } + }; + let tree = walk("/p", &[], 15_000, lister).expect("the walk succeeds"); + assert_eq!( + render(&tree), + "/p/\n - file\n - link\n - sub/\n - inner" + ); +} + +#[test] +fn every_other_kind_of_node_is_listed_as_a_plain_name() { + let lister = |_: &str| -> Result, VfsError> { + Ok(vec![ + entry("block", FileType::BlockDevice), + entry("char", FileType::CharDevice), + entry("fifo", FileType::Fifo), + entry("sock", FileType::Socket), + ]) + }; + let tree = walk("/dev", &[], 15_000, lister).expect("the walk succeeds"); + assert_eq!( + render(&tree), + "/dev/\n - block\n - char\n - fifo\n - sock" + ); +} + +#[test] +fn the_pack_sorts_each_listing_itself_because_list_promises_no_order() { + let lister = |path: &str| -> Result, VfsError> { + match path { + "/p" => Ok(vec![ + entry("b", FileType::File), + entry("a", FileType::Directory), + entry("B", FileType::File), + entry("a-b", FileType::File), + ]), + "/p/a" => Ok(vec![entry("z", FileType::File), entry("Z", FileType::File)]), + other => panic!("the walk entered {other}"), + } + }; + let tree = walk("/p", &[], 15_000, lister).expect("the walk succeeds"); + assert_eq!( + render(&tree), + "/p/\n - B\n - a-b\n - a/\n - Z\n - z\n - b" + ); +} + +#[test] +fn a_failure_listing_the_root_is_returned_for_the_tool_to_word() { + let lister = |path: &str| -> Result, VfsError> { + Err(VfsError::NotADirectory { + path: path.to_owned(), + }) + }; + let failure = walk("/link/", &[], 15_000, lister).expect_err("the root cannot be listed"); + assert_eq!( + failure, + Failure { + path: "/link/".to_owned(), + error: VfsError::NotADirectory { + path: "/link".to_owned() + }, + }, + "the root's path is the target exactly as written" + ); +} + +#[test] +fn a_subdirectory_that_is_gone_or_no_longer_a_directory_is_shown_without_children() { + let lister = |path: &str| -> Result, VfsError> { + match path { + "/p" => Ok(vec![ + entry("gone", FileType::Directory), + entry("open", FileType::Directory), + entry("swapped", FileType::Directory), + ]), + "/p/gone" => Err(VfsError::NotFound { + path: path.to_owned(), + }), + "/p/open" => Ok(vec![entry("a", FileType::File)]), + "/p/swapped" => Err(VfsError::NotADirectory { + path: path.to_owned(), + }), + other => panic!("the walk entered {other}"), + } + }; + let tree = walk("/p", &[], 15_000, lister).expect("an empty look at these is true"); + assert_eq!( + render(&tree), + "/p/\n - gone/\n - open/\n - a\n - swapped/" + ); +} + +#[test] +fn a_subdirectory_that_cannot_be_listed_for_any_other_reason_fails_the_walk() { + for label in ["conflict", "denied", "io"] { + let lister = |path: &str| -> Result, VfsError> { + match path { + "/p" => Ok(vec![ + entry("locked", FileType::Directory), + entry("open", FileType::Directory), + ]), + "/p/locked" => Err(listing_error(label, path)), + "/p/open" => Ok(vec![entry("a", FileType::File)]), + other => panic!("the walk entered {other}"), + } + }; + let failure = walk("/p", &[], 15_000, lister).expect_err(label); + assert_eq!( + failure.path, "/p/locked", + "{label}: the directory that failed" + ); + assert_eq!(failure.error, listing_error(label, "/p/locked"), "{label}"); + } +} + +/// The error a listing of `path` fails with, for the kind named by `label`. +fn listing_error(label: &str, path: &str) -> VfsError { + match label { + "conflict" => VfsError::Conflict { + path: path.to_owned(), + detail: "another call holds a write claim".to_owned(), + }, + "denied" => VfsError::PermissionDenied { + path: path.to_owned(), + reason: "Reading /p/locked is denied by policy.".to_owned(), + }, + "io" => VfsError::Io { + path: path.to_owned(), + kind: std::io::ErrorKind::PermissionDenied, + raw_os_error: None, + message: "access denied".to_owned(), + }, + other => panic!("no listing error named {other}"), + } +} + +#[test] +fn a_conflict_below_the_root_is_worded_for_the_directory_that_conflicted() { + let lister = |path: &str| -> Result, VfsError> { + match path { + "notes" => Ok(vec![entry("locked", FileType::Directory)]), + "notes/locked" => Err(VfsError::Conflict { + path: "/notes/locked".to_owned(), + detail: "another call holds a write claim".to_owned(), + }), + other => panic!("the walk entered {other}"), + } + }; + let failure = walk("notes", &[], 15_000, lister).expect_err("the conflict fails the walk"); + assert_eq!( + refusal("ListDir", &failure.path, &failure.error), + "notes/locked was changed, or is being changed, by another call in this round, so it \ + cannot be read in this round. Call ListDir again in your next message." + ); +} + +#[test] +fn every_successful_listing_is_untrusted_output() { + let call = seeded(&["/p/a.md", "/p/.git/HEAD"]); + call.context() + .access() + .mkdir("/empty", false) + .expect("the directory is made"); + for args in [ + json!({ "target_directory": "/p" }), + json!({ "target_directory": "/empty" }), + json!({ "target_directory": "/" }), + json!({ "target_directory": "/p", "ignore_globs": ["*.md"] }), + ] { + let output = list(&call, &args).expect("the listing succeeds"); + assert_eq!(output.trust(), OutputTrust::Untrusted, "{args}"); + } +} diff --git a/crates/plugin-std/src/dir/walk.rs b/crates/plugin-std/src/dir/walk.rs new file mode 100644 index 000000000..62fd00f91 --- /dev/null +++ b/crates/plugin-std/src/dir/walk.rs @@ -0,0 +1,234 @@ +//! The walk: lists a directory tree breadth first and stops at the byte +//! cap, so a huge tree is never read whole. +//! +//! The walk returns a [`Tree`] and writes no text. Each directory's +//! entries are sorted in the shared walk order (`crate::order`), because +//! `Access::list` promises no order. That order equals the order +//! `Access::glob` gives. + +use std::collections::VecDeque; + +use runvil_plugin::{Entry, FileType, VfsError}; + +use crate::matcher::Matcher; +use crate::order::{GIT, sorted}; + +/// One line of the tree: a name under its parent. +#[derive(Debug)] +pub(super) struct Node { + /// The entry's name. The root's is empty. + pub(super) name: String, + /// Whether the entry is a directory, which ends in `/` and is entered. + /// A link and every other kind of node is not. + pub(super) directory: bool, + /// Whether the walk got as far as showing this directory's contents. + /// It is `false` only for a directory that was cut off before its + /// first child was shown, and it prints with `...` after its name. + pub(super) expanded: bool, + /// The indexes in [`Tree::nodes`] of the entries shown under this one, + /// in sorted order. + pub(super) children: Vec, +} + +/// The tree the walk found. +#[derive(Debug)] +pub(super) struct Tree { + /// The root's line: its absolute virtual path, ending in `/`. + pub(super) root_line: String, + /// Every node shown. The first is the root. + pub(super) nodes: Vec, + /// Whether the walk stopped because the next line would pass the cap. + pub(super) truncated: bool, +} + +/// A listing that failed and ended the walk. +#[derive(Debug, PartialEq, Eq)] +pub(super) struct Failure { + /// The directory that could not be listed, written as the model wrote + /// the target with the names below it added. For the root it is the + /// target exactly as written. + pub(super) path: String, + /// Why the listing failed. + pub(super) error: VfsError, +} + +/// A directory the walk still has to list. +struct Pending { + /// Its index in the tree's nodes. + node: usize, + /// The path the lister is given for it. + path: String, + /// The depth of the entries under it: 1 for the root's. + depth: usize, +} + +/// Walks the directory `target`, breadth first, calling `list` for each +/// directory it enters. +/// +/// `target` is the path as the model wrote it. `ignore` leaves out each +/// entry whose absolute virtual path a pattern matches, and a left-out +/// directory is not entered. The walk counts the bytes of every line it +/// adds, the root's included, each with its newline, and when the next line +/// would pass `cap` it stops everything: the directory being listed keeps +/// the children already added, and every directory shown but not expanded +/// is marked. +/// +/// # Errors +/// Returns the lister's error, with the directory it was for, when the +/// root cannot be listed, and for a directory below the root when the +/// failure is anything but the directory being gone or no longer a +/// directory. A claim conflict or a refusal by policy is returned rather +/// than shown as an empty directory, because the model would read +/// `locked/` as empty when its contents were never seen. Only a directory +/// that is gone, or is no longer a directory, is shown with no children, +/// because an empty look at it is true. +pub(super) fn walk( + target: &str, + ignore: &[Matcher], + cap: usize, + mut list: F, +) -> Result +where + F: FnMut(&str) -> Result, VfsError>, +{ + let root_line = root_line(target); + let mut used = root_line.len() + 1; + let mut nodes = vec![Node { + name: String::new(), + directory: true, + expanded: true, + children: Vec::new(), + }]; + let mut queue = VecDeque::from([Pending { + node: 0, + path: directory_path(target), + depth: 1, + }]); + let mut truncated = false; + while let Some(pending) = queue.pop_front() { + let entries = match list(&pending.path) { + Ok(entries) => entries, + Err(error) if pending.node != 0 && is_gone(&error) => Vec::new(), + Err(error) => { + let path = if pending.node == 0 { + target.to_owned() + } else { + pending.path + }; + return Err(Failure { path, error }); + } + }; + for entry in visible(entries, &pending.path, ignore) { + let directory = matches!(entry.stat.file_type, FileType::Directory); + let cost = line_cost(pending.depth, &entry.name, directory); + if used + cost > cap { + truncated = true; + break; + } + used += cost; + let index = nodes.len(); + if directory { + queue.push_back(Pending { + node: index, + path: join(&pending.path, &entry.name), + depth: pending.depth + 1, + }); + } + nodes.push(Node { + name: entry.name, + directory, + expanded: !directory, + children: Vec::new(), + }); + nodes[pending.node].children.push(index); + } + let listed = &mut nodes[pending.node]; + listed.expanded = !truncated || !listed.children.is_empty(); + if truncated { + break; + } + } + Ok(Tree { + root_line, + nodes, + truncated, + }) +} + +/// Whether `error` from listing a directory below the root means the +/// directory is gone, or is no longer a directory, so showing it with no +/// children is true. +fn is_gone(error: &VfsError) -> bool { + matches!( + error, + VfsError::NotFound { .. } | VfsError::NotADirectory { .. } + ) +} + +/// The entries of one directory the tree shows, in order: everything but +/// `.git` and what `ignore` matches, in the shared walk order. +fn visible(entries: Vec, directory: &str, ignore: &[Matcher]) -> Vec { + let kept: Vec = entries + .into_iter() + .filter(|entry| entry.name != GIT && !is_ignored(directory, &entry.name, ignore)) + .collect(); + sorted(kept) +} + +/// Whether a pattern in `ignore` matches the entry `name` of `directory`, +/// by the entry's absolute virtual path, which has no trailing `/` even +/// for a directory. +fn is_ignored(directory: &str, name: &str, ignore: &[Matcher]) -> bool { + if ignore.is_empty() { + return false; + } + let path = virtual_path(&join(directory, name)); + ignore.iter().any(|matcher| matcher.is_match(&path)) +} + +/// The bytes a line costs against the cap: the indent of two spaces per +/// level, the bullet, the name, the `/` of a directory, and the newline. +fn line_cost(depth: usize, name: &str, directory: bool) -> usize { + 2 * depth + "- ".len() + name.len() + usize::from(directory) + 1 +} + +/// `name` inside `directory`, with one `/` between them. +fn join(directory: &str, name: &str) -> String { + if directory.ends_with('/') { + format!("{directory}{name}") + } else { + format!("{directory}/{name}") + } +} + +/// The path the lister is given for the root: the target as written, +/// without a trailing `/`, and `/` alone for the filesystem root. +fn directory_path(target: &str) -> String { + let trimmed = target.trim_end_matches('/'); + if trimmed.is_empty() { + "/".to_owned() + } else { + trimmed.to_owned() + } +} + +/// `path` with a leading `/` added when it lacks one. No canonical path is +/// available, so this is as far as the model's path is turned into an +/// absolute one. +fn virtual_path(path: &str) -> String { + if path.starts_with('/') { + path.to_owned() + } else { + format!("/{path}") + } +} + +/// The root's line: the target as an absolute virtual path, with `/` +/// appended unless it already ends in one. +fn root_line(target: &str) -> String { + let mut line = virtual_path(target); + if !line.ends_with('/') { + line.push('/'); + } + line +} diff --git a/crates/plugin-std/src/grep.rs b/crates/plugin-std/src/grep.rs new file mode 100644 index 000000000..81c3ea024 --- /dev/null +++ b/crates/plugin-std/src/grep.rs @@ -0,0 +1,62 @@ +//! The `Grep` tool: a regular-expression search over the files under a +//! path. +//! +//! Grep walks the tree through the call's [`Access`], so the run's mode +//! and policy apply, and reads each file whole, never more than 16 MiB of +//! it. Files come in path order, the order Glob uses. Output stops at +//! 15,000 bytes, and every call runs from the start, so a call with an +//! `offset` pages through the same result. An entry that cannot be +//! searched is skipped and counted, never a failure. The text of the +//! files is returned as untrusted output. + +mod args; +mod find; +mod found; +mod matching; +mod pause; +mod render; +mod source; +mod types; +mod window; + +#[cfg(test)] +mod tests; + +use runvil_plugin::{Access, ToolError, ToolOutput}; +use serde_json::Value; + +use crate::spec::Definition; + +use self::args::Request; +use self::found::Limits; +use self::source::Source; + +/// Runs one `Grep` call whose arguments already passed the shared schema +/// check. +/// +/// # Errors +/// Returns an invalid-arguments [`ToolError`] naming the parameter for a +/// bad argument, and a message error for a root that cannot be searched, +/// worded for a model to read. +pub(crate) async fn call( + access: &Access, + definition: &Definition, + args: &Value, +) -> Result { + let request = args::parse(definition, args)?; + search(access, &definition.wire_name, &request, &Limits::default()).await +} + +/// Searches `source` for what `request` asks, under `limits`. `tool` is +/// the name the model called the tool by. +async fn search( + source: &S, + tool: &str, + request: &Request<'_>, + limits: &Limits, +) -> Result { + let found = find::find(source, tool, request, limits) + .await + .map_err(ToolError::message)?; + Ok(render::render(&found, request, limits)) +} diff --git a/crates/plugin-std/src/grep/args.rs b/crates/plugin-std/src/grep/args.rs new file mode 100644 index 000000000..ac52bfce5 --- /dev/null +++ b/crates/plugin-std/src/grep/args.rs @@ -0,0 +1,262 @@ +//! Grep's arguments: what the call asks for, and what is refused before +//! any path is looked at. + +use regex::{Regex, RegexBuilder}; +use runvil_plugin::ToolError; +use serde_json::Value; + +use super::types; +use crate::check::{invalid_arguments, invalid_pattern, refuse_blank, refuse_unknown}; +use crate::matcher::Matcher; +use crate::spec::Definition; + +/// What Grep prints for each match. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum Mode { + /// The matching lines, with their context. + Content, + /// The paths of the files that match. + FilesWithMatches, + /// Each matching file's path and its count of matching lines. + Count, +} + +/// What a call asks Grep for. +#[derive(Debug)] +pub(super) struct Request<'a> { + /// The file or directory to search, as the model wrote it. `None` + /// means the filesystem root. + pub(super) path: Option<&'a str>, + /// The `glob` filter as the model wrote it, and compiled. + pub(super) glob: Option<(&'a str, Matcher)>, + /// The extensions the `type` filter keeps. + pub(super) kind: Option<&'static [&'static str]>, + /// What to print for each match. + pub(super) mode: Mode, + /// The context lines shown before each match, in `content` mode. + pub(super) before: usize, + /// The context lines shown after each match, in `content` mode. + pub(super) after: usize, + /// The most entries to show, or `None` for no limit. + pub(super) limit: Option, + /// The entries to skip first. + pub(super) offset: usize, + /// Whether a match may span lines, so the whole text is one subject. + pub(super) multiline: bool, + /// The compiled pattern. + pub(super) regex: Regex, +} + +/// The one key a call may pass that the schema does not declare: Cursor's +/// `-n`, which a model may still send, and which is accepted when it is +/// `true`. +const ACCEPTED_UNDECLARED: &str = "-n"; + +/// Reads the call's arguments after the shared schema check. +/// +/// The order is fixed, so a call that breaks two rules always hears about +/// the same one: an undeclared key (the one undeclared key `-n` is +/// accepted), the `-n` value, an empty pattern, a negative `-A`, `-B`, or +/// `-C`, a negative `head_limit` or `offset`, an unknown `type`, an +/// invalid `glob`, a blank `path`, and last the pattern itself, which is +/// compiled before any filesystem call. +/// +/// # Errors +/// Returns an invalid-arguments [`ToolError`] naming the parameter. +pub(super) fn parse<'a>( + definition: &Definition, + args: &'a Value, +) -> Result, ToolError> { + let tool = definition.wire_name.as_str(); + let mut declared = args.clone(); + if let Some(object) = declared.as_object_mut() { + object.remove(ACCEPTED_UNDECLARED); + } + refuse_unknown(tool, &definition.parameters, &declared)?; + refuse_line_numbers_off(tool, args)?; + let pattern = args + .get("pattern") + .and_then(Value::as_str) + .unwrap_or_default(); + if pattern.is_empty() { + return Err(invalid_arguments( + tool, + "`pattern` is empty. Pass \".\" to match every line.", + )); + } + let after = number(tool, args, "-A", "context lines")?; + let before = number(tool, args, "-B", "context lines")?; + let both = number(tool, args, "-C", "context lines")?; + let limit = number(tool, args, "head_limit", "")?.map(|value| count(value, true)); + let offset = number(tool, args, "offset", "")?.map(|value| count(value, false)); + let kind = kind(tool, args)?; + let glob = glob(tool, args)?; + let path = path(tool, args)?; + let mode = mode(args); + let (before, after) = if mode == Mode::Content { + ( + before.or(both).map_or(0, |value| count(value, true)), + after.or(both).map_or(0, |value| count(value, true)), + ) + } else { + (0, 0) + }; + let multiline = flag(args, "multiline"); + let regex = compile(tool, pattern, args, multiline)?; + Ok(Request { + path, + glob, + kind, + mode, + before, + after, + limit: limit.filter(|limit| *limit > 0), + offset: offset.unwrap_or_default(), + multiline, + regex, + }) +} + +/// Refuses `-n` unless it is `true` or left out, because Grep always shows +/// line numbers. +fn refuse_line_numbers_off(tool: &str, args: &Value) -> Result<(), ToolError> { + match args.get(ACCEPTED_UNDECLARED) { + None | Some(Value::Null | Value::Bool(true)) => Ok(()), + Some(_) => Err(invalid_arguments( + tool, + "`-n` must be true or left out, because line numbers are always shown.", + )), + } +} + +/// The number under `name`, or `None` when the call left it out or passed +/// `null`, refused when it is negative. +/// +/// `unit` names what a context line count counts, and is empty for +/// `head_limit` and `offset`, whose refusal is shorter. +fn number(tool: &str, args: &Value, name: &str, unit: &str) -> Result, ToolError> { + let Some(value) = args.get(name).filter(|value| !value.is_null()) else { + return Ok(None); + }; + let Some(number) = value.as_f64() else { + return Ok(None); + }; + if number < 0.0 { + let ask = if unit.is_empty() { + "Pass 0 or more.".to_owned() + } else { + format!("Pass 0 or more {unit}.") + }; + return Err(invalid_arguments( + tool, + format!("`{name}` is {value}. {ask}"), + )); + } + Ok(Some(number)) +} + +/// The whole number `value` rounds to, up or down. A value past the range +/// of `usize` is the largest `usize`. +#[expect( + clippy::cast_possible_truncation, + clippy::cast_sign_loss, + reason = "the value is not negative, and a float to integer cast saturates at the largest usize" +)] +fn count(value: f64, up: bool) -> usize { + let whole = if up { value.ceil() } else { value.floor() }; + whole as usize +} + +/// The extensions of the `type` the call names, or `None` when it names +/// none. An empty string names none. +fn kind(tool: &str, args: &Value) -> Result, ToolError> { + let Some(name) = text(args, "type") else { + return Ok(None); + }; + types::extensions(name).map(Some).ok_or_else(|| { + invalid_arguments( + tool, + format!( + "Unrecognized file type \"{name}\". Use a type such as js, ts, py, rust, go or \ + java, or filter with glob." + ), + ) + }) +} + +/// The `glob` the call names, compiled, or `None` when it names none. +fn glob<'a>(tool: &str, args: &'a Value) -> Result, ToolError> { + let Some(pattern) = text(args, "glob") else { + return Ok(None); + }; + Matcher::new(pattern) + .map(|matcher| Some((pattern, matcher))) + .map_err(|error| invalid_pattern(tool, "glob", pattern, error)) +} + +/// The `path` the call names, or `None` when it names none, refused when +/// it is only whitespace. +fn path<'a>(tool: &str, args: &'a Value) -> Result, ToolError> { + let Some(path) = text(args, "path") else { + return Ok(None); + }; + refuse_blank(tool, "path", path)?; + Ok(Some(path)) +} + +/// The string under `name`, or `None` when the call left it out, passed +/// `null`, or passed an empty string. +fn text<'a>(args: &'a Value, name: &str) -> Option<&'a str> { + args.get(name) + .and_then(Value::as_str) + .filter(|text| !text.is_empty()) +} + +/// The output mode: `content` unless the call names another. +fn mode(args: &Value) -> Mode { + match args.get("output_mode").and_then(Value::as_str) { + Some("files_with_matches") => Mode::FilesWithMatches, + Some("count") => Mode::Count, + _ => Mode::Content, + } +} + +/// The boolean under `name`, `false` when the call left it out. +fn flag(args: &Value, name: &str) -> bool { + args.get(name).and_then(Value::as_bool).unwrap_or(false) +} + +/// Compiles the pattern with the call's flags. +/// +/// Matching is case-sensitive unless `-i` is set, and `^` and `$` match at +/// each line. With `multiline`, `.` also matches a line break and a line +/// break is `\r\n` or `\n`. `literal` matches the pattern as plain text. +fn compile(tool: &str, pattern: &str, args: &Value, multiline: bool) -> Result { + let source = if flag(args, "literal") { + regex::escape(pattern) + } else { + pattern.to_owned() + }; + RegexBuilder::new(&source) + .case_insensitive(flag(args, "-i")) + .multi_line(true) + .dot_matches_new_line(multiline) + .crlf(multiline) + .build() + .map_err(|error| { + // The crate's syntax errors echo the pattern and then end with + // `error: `, so the reason is what follows the last one. + let message = error.to_string(); + let reason = message + .rsplit_once("error: ") + .map_or(message.trim(), |(_, reason)| reason.trim()); + invalid_arguments( + tool, + format!( + "Invalid regular expression \"{pattern}\": {reason}. Grep matches with Rust \ + regex syntax, which has no look-around and no back-references. To match the \ + text literally, set literal to true." + ), + ) + }) +} diff --git a/crates/plugin-std/src/grep/find.rs b/crates/plugin-std/src/grep/find.rs new file mode 100644 index 000000000..f73ce7b2a --- /dev/null +++ b/crates/plugin-std/src/grep/find.rs @@ -0,0 +1,388 @@ +//! The search: walks the tree, reads each file, and gathers the entries +//! the answer shows. +//! +//! [`find`] is a pure function of its [`Source`]: it reads no clock, keeps +//! nothing between calls, and returns what it found without writing any +//! text. It yields to the executor now and then, so a running search can +//! be cancelled. + +mod collect; + +use runvil_plugin::{Entry, FileType, VfsError}; + +use super::args::{Mode, Request}; +use super::found::{Found, Limits, Reason, Stop}; +use super::matching::each_match_line; +use super::pause::Pause; +use super::source::Source; +use super::types; +use crate::order::{GIT, sorted}; +use crate::refusal::{missing_root, neither_file_nor_directory, refusal}; +use crate::text::{decode, sniff_binary, split_lines}; + +/// A path as the file operations are given it, beside the absolute path +/// the answer prints. +/// +/// The two differ only when the model wrote a path with no leading `/`: +/// the operations get it as written, and the answer adds the `/`. +struct Node { + /// The path to pass to the [`Source`]. + access: String, + /// The absolute virtual path to print and to match a glob against. + shown: String, +} + +impl Node { + /// The node for the directory the model named, without a trailing `/`. + fn directory(path: &str) -> Node { + let trimmed = path.trim_end_matches('/'); + let access = if trimmed.is_empty() { + "/".to_owned() + } else { + trimmed.to_owned() + }; + let shown = if access.starts_with('/') { + access.clone() + } else { + format!("/{access}") + }; + Node { access, shown } + } + + /// The node for the file the model named. + fn file(path: &str) -> Node { + let shown = if path.starts_with('/') { + path.to_owned() + } else { + format!("/{path}") + }; + Node { + access: path.to_owned(), + shown, + } + } + + /// The node for the entry `name` of this directory. + fn join(&self, name: &str) -> Node { + let slash = |path: &str| { + if path.ends_with('/') { + format!("{path}{name}") + } else { + format!("{path}/{name}") + } + }; + Node { + access: slash(&self.access), + shown: slash(&self.shown), + } + } +} + +/// A directory the walk is part way through. +struct Frame { + /// The directory. + dir: Node, + /// Its entries not yet visited, in the order they are visited. + entries: std::vec::IntoIter, +} + +/// Searches the tree at the request's path. +/// +/// `tool` is the name the model called the tool by. +/// +/// # Errors +/// Returns the sentence the model reads when the root cannot be searched: +/// it does not exist, it cannot be listed or read, or it is neither a +/// file nor a directory. Nothing below the root fails the search. +pub(super) async fn find( + source: &S, + tool: &str, + request: &Request<'_>, + limits: &Limits, +) -> Result { + let root = request.path.unwrap_or("/"); + let stat = match source.stat(root) { + Ok(stat) => stat, + Err(VfsError::NotFound { .. }) => return Err(missing_root(root)), + Err(error) => return Err(refusal(tool, root, &error)), + }; + let mut search = Search { + source, + tool, + request, + limits, + found: Found::default(), + cost: 0, + considered: 0, + bytes: 0, + since_files: 0, + since_bytes: 0, + finished: false, + }; + match stat.file_type { + FileType::Directory => search.directory(root).await?, + FileType::File => search.root_file(root, Some(stat.size))?, + FileType::Symlink => search.link(root).await?, + _ => return Err(neither_file_nor_directory(root)), + } + Ok(search.found) +} + +/// One search in progress. +struct Search<'a, 'r, S: Source> { + /// Where the tree is read from. + source: &'a S, + /// The name the model called the tool by. + tool: &'a str, + /// What the call asked for. + request: &'a Request<'r>, + /// The limits the search runs under. + limits: &'a Limits, + /// What has been found so far. + found: Found, + /// The bytes the kept lines will take in the answer, counted the way + /// the render counts them. + cost: usize, + /// How many files have been considered. + considered: usize, + /// How many bytes have been read. + bytes: u64, + /// The files considered since the last pause. + since_files: usize, + /// The bytes read since the last pause. + since_bytes: u64, + /// Whether the search has everything it needs, or hit a limit. + finished: bool, +} + +impl Search<'_, '_, S> { + /// Searches the directory the model named. + async fn directory(&mut self, root: &str) -> Result<(), String> { + let node = Node::directory(root); + let entries = self + .source + .list(&node.access) + .map_err(|error| refusal(self.tool, root, &error))?; + self.walk(node, entries).await; + Ok(()) + } + + /// Searches the link the model named: as a directory when it lists, + /// and as a file when it does not. + async fn link(&mut self, root: &str) -> Result<(), String> { + let node = Node::directory(root); + match self.source.list(&node.access) { + Ok(entries) => { + self.walk(node, entries).await; + Ok(()) + } + Err(VfsError::NotADirectory { .. }) => self.root_file(root, None), + Err(error) => Err(refusal(self.tool, root, &error)), + } + } + + /// Searches the one file the model named. No ignore rule, glob, or + /// type applies, because the model named it. + /// + /// `size` is the file's size when the stat gave it, and `None` for a + /// link, whose own size says nothing about its target. + fn root_file(&mut self, root: &str, size: Option) -> Result<(), String> { + let node = Node::file(root); + self.considered += 1; + let bytes = if let Some(size) = size { + if size > self.limits.max_file_size { + self.skip(Reason::OverSize, Some(&node.shown)); + return Ok(()); + } + self.source.read(&node.access) + } else { + self.source + .read_prefix(&node.access, self.limits.max_file_size.saturating_add(1)) + } + .map_err(|error| refusal(self.tool, root, &error))?; + if size.is_none() && length(&bytes) > self.limits.max_file_size { + self.skip(Reason::OverSize, Some(&node.shown)); + return Ok(()); + } + self.bytes += length(&bytes); + self.consume(&node.shown, &bytes); + Ok(()) + } + + /// Walks the directory `root`, whose `entries` are already listed, + /// depth first. + async fn walk(&mut self, root: Node, entries: Vec) { + let mut stack = vec![Frame { + dir: root, + entries: sorted(entries).into_iter(), + }]; + while let Some(frame) = stack.last_mut() { + if self.finished { + return; + } + let Some(entry) = frame.entries.next() else { + stack.pop(); + continue; + }; + if entry.name == GIT { + continue; + } + let child = frame.dir.join(&entry.name); + if entry.name.chars().any(|c| c < ' ' || c == '\u{7f}') { + self.skip(Reason::ControlName, None); + continue; + } + match entry.stat.file_type { + FileType::Symlink => self.skip(Reason::SymbolicLink, Some(&child.shown)), + FileType::Directory => match self.source.list(&child.access) { + Ok(entries) => stack.push(Frame { + dir: child, + entries: sorted(entries).into_iter(), + }), + Err(error) => self.skip(directory_reason(&error), Some(&child.shown)), + }, + // A pipe, a socket, or a device is never opened: reading one + // can block for good, so it is skipped as unreadable. + FileType::File => { + self.visit(&child, &entry.name, entry.stat.size); + self.pace().await; + } + _ => self.skip(Reason::UnreadableFile, Some(&child.shown)), + } + } + } + + /// Visits one file the walk met: counts it, applies the filters, + /// reads it, and searches it. + fn visit(&mut self, node: &Node, name: &str, size: u64) { + if self.considered >= self.limits.max_files { + self.finish(Stop::Files); + return; + } + self.considered += 1; + self.since_files += 1; + if let Some((_, matcher)) = &self.request.glob + && !matcher.is_match(&node.shown) + { + return; + } + if let Some(extensions) = self.request.kind + && !types::matches(extensions, name) + { + return; + } + if size > self.limits.max_file_size { + self.skip(Reason::OverSize, Some(&node.shown)); + return; + } + if self.bytes >= self.limits.max_bytes { + self.finish(Stop::Bytes); + return; + } + match self.source.read(&node.access) { + Ok(bytes) => { + self.bytes += length(&bytes); + self.since_bytes += length(&bytes); + self.consume(&node.shown, &bytes); + } + Err(error) => self.skip(file_reason(&error), Some(&node.shown)), + } + } + + /// Pauses when enough files or bytes have gone by since the last + /// pause, so a cancel or a stop can drop the future. + async fn pace(&mut self) { + if self.since_files >= self.limits.slice_files + || self.since_bytes >= self.limits.slice_bytes + { + self.since_files = 0; + self.since_bytes = 0; + Pause::default().await; + } + } + + /// Counts one skipped entry, and keeps its path for the skip line + /// when it has a printable one. + fn skip(&mut self, reason: Reason, path: Option<&str>) { + self.found.skips.record(reason, path); + } + + /// Ends the search at a limit. + fn finish(&mut self, stop: Stop) { + self.found.stop = Some(stop); + self.finished = true; + } + + /// Searches the bytes of the file at `path`. + fn consume(&mut self, path: &str, bytes: &[u8]) { + if sniff_binary(bytes) { + self.skip(Reason::Binary, Some(path)); + return; + } + let decoded = decode(bytes); + self.found.searched += 1; + if decoded.invalid > 0 { + self.found.replaced.record(path); + } + let text = decoded.text; + let lines = split_lines(&text); + match self.request.mode { + Mode::Content => self.collect_lines(path, &text, &lines), + Mode::FilesWithMatches => { + let mut matched = false; + each_match_line( + &self.request.regex, + self.request.multiline, + &text, + &lines, + |_, _| { + matched = true; + false + }, + ); + if matched { + self.collect_file(path, 0); + } + } + Mode::Count => { + let mut matching = 0; + each_match_line( + &self.request.regex, + self.request.multiline, + &text, + &lines, + |_, _| { + matching += 1; + true + }, + ); + if matching > 0 { + self.collect_file(path, matching); + } + } + } + } +} + +/// The length of `bytes` as a `u64`. +fn length(bytes: &[u8]) -> u64 { + u64::try_from(bytes.len()).unwrap_or(u64::MAX) +} + +/// Why a directory that failed to list is skipped. +fn directory_reason(error: &VfsError) -> Reason { + match error { + VfsError::Conflict { .. } => Reason::Conflict, + VfsError::PermissionDenied { .. } | VfsError::ApprovalRequired { .. } => Reason::Denied, + _ => Reason::UnreadableDirectory, + } +} + +/// Why a file that failed to read is skipped. +fn file_reason(error: &VfsError) -> Reason { + match error { + VfsError::Conflict { .. } => Reason::Conflict, + VfsError::PermissionDenied { .. } | VfsError::ApprovalRequired { .. } => Reason::Denied, + _ => Reason::UnreadableFile, + } +} diff --git a/crates/plugin-std/src/grep/find/collect.rs b/crates/plugin-std/src/grep/find/collect.rs new file mode 100644 index 000000000..911c3df32 --- /dev/null +++ b/crates/plugin-std/src/grep/find/collect.rs @@ -0,0 +1,157 @@ +//! Gathering the entries the answer shows: which entries are kept, and +//! the lines each one keeps. +//! +//! Entries are counted in walk order across all files. The first `offset` +//! are only counted, the next `head_limit` are kept, and one more is +//! counted to show the head limit was reached. A kept line is stored as +//! the text the answer will show, so no decoded file is held after it is +//! searched. + +use std::collections::BTreeMap; + +use super::Search; +use crate::grep::found::{FileHit, Row}; +use crate::grep::matching::each_match_line; +use crate::grep::render::{GAP_COST, row_cost, summary_cost}; +use crate::grep::source::Source; +use crate::grep::window::{context_text, match_text}; + +/// What to do with the next entry. +#[derive(Debug, PartialEq, Eq)] +enum Admission { + /// It comes before the offset: it is counted and nothing more. + Skip, + /// It is kept. + Keep, + /// It is the one past the head limit, which is counted and not kept. + Extra, +} + +impl Search<'_, '_, S> { + /// Decides what the next entry is: one before the offset that is only + /// counted, one to keep, or the extra one that shows the head limit + /// was reached. + fn admit(&mut self) -> Admission { + let index = self.found.seen; + self.found.seen += 1; + let offset = self.request.offset; + if index < offset { + return Admission::Skip; + } + if self + .request + .limit + .is_some_and(|limit| index - offset >= limit) + { + self.found.extra = true; + self.found.collected += 1; + self.finished = true; + return Admission::Extra; + } + self.found.collected += 1; + Admission::Keep + } + + /// Ends the search when the kept lines already fill the body cap, so + /// the entries past them, which are never shown, are not gathered. + fn check_cap(&mut self) { + if self.cost > self.limits.body_cap { + self.found.capped = true; + self.finished = true; + } + } + + /// Gathers one file's entry for `files_with_matches` or `count`. + pub(super) fn collect_file(&mut self, path: &str, count: usize) { + if self.admit() != Admission::Keep { + return; + } + self.cost += summary_cost(self.request.mode, path, count); + self.found.hits.push(FileHit { + path: path.to_owned(), + count, + rows: BTreeMap::new(), + }); + self.check_cap(); + } + + /// Gathers the matching lines of one file for `content` mode. + pub(super) fn collect_lines(&mut self, path: &str, text: &str, lines: &[&str]) { + let mut hit: Option = None; + let request = self.request; + each_match_line( + &request.regex, + request.multiline, + text, + lines, + |line, first| match self.admit() { + Admission::Skip => true, + Admission::Extra => false, + Admission::Keep => { + let hit = hit.get_or_insert_with(|| self.begin(path)); + self.keep(hit, lines, line, first); + self.check_cap(); + !self.finished + } + }, + ); + if let Some(hit) = hit { + self.found.hits.push(hit); + } + } + + /// Starts the result of the file at `path`, and counts the lines its + /// path and the blank line before it take. + fn begin(&mut self, path: &str) -> FileHit { + self.cost += path.len() + 1 + usize::from(!self.found.hits.is_empty()); + FileHit { + path: path.to_owned(), + count: 0, + rows: BTreeMap::new(), + } + } + + /// Keeps the matching line at index `line` of `lines`, whose first + /// match starts at byte `first`, with the context lines around it. + /// + /// Lines are kept in order, and stop being kept once the cost passes + /// the body cap, because no line past that point is shown. + fn keep(&mut self, hit: &mut FileHit, lines: &[&str], line: usize, first: usize) { + let request = self.request; + let window = self.limits.window; + let from = line.saturating_sub(request.before); + let to = line + .saturating_add(request.after) + .min(lines.len().saturating_sub(1)); + for index in from..=to { + if self.cost > self.limits.body_cap { + break; + } + let number = index + 1; + let is_match = index == line; + if !is_match && hit.rows.contains_key(&number) { + continue; + } + let Some(source) = lines.get(index) else { + break; + }; + let text = if is_match { + match_text(source, first, &window) + } else { + context_text(source, &window) + }; + self.cost += row_cost(number, &text); + if let Some(old) = hit.rows.insert(number, Row { text, is_match }) { + self.cost -= row_cost(number, &old.text); + } else if (request.before > 0 || request.after > 0) + && hit + .rows + .range(..number) + .next_back() + .is_some_and(|(previous, _)| number > previous + 1) + { + self.cost += GAP_COST; + } + } + } +} diff --git a/crates/plugin-std/src/grep/found.rs b/crates/plugin-std/src/grep/found.rs new file mode 100644 index 000000000..e094a1bc9 --- /dev/null +++ b/crates/plugin-std/src/grep/found.rs @@ -0,0 +1,204 @@ +//! What a search finds and the limits it runs under. +//! +//! [`find`](super::find::find) fills a [`Found`] and writes no text, and +//! [`render`](super::render::render) turns it into the answer. + +use std::collections::BTreeMap; + +use super::window::Window; +use crate::text::{BODY_CAP, MAX_FILE_BYTES}; + +/// The most files one search considers. +pub(super) const MAX_FILES: usize = 100_000; + +/// The most bytes one search reads: 1 GiB. +const MAX_BYTES: u64 = 1_073_741_824; + +/// How many files are considered between two pauses. +const SLICE_FILES: usize = 32; + +/// How many bytes are read between two pauses: 2 MiB. +const SLICE_BYTES: u64 = 2_097_152; + +/// How many skipped paths the skip line names. +const FIRST_KEPT: usize = 5; + +/// The limits a search runs under. Each is a parameter, with the +/// constant as its default, so a test can lower it. +#[derive(Debug, Clone, Copy)] +pub(super) struct Limits { + /// The most files considered before the search stops. + pub(super) max_files: usize, + /// The most bytes read before the search stops. + pub(super) max_bytes: u64, + /// The largest file, by the size its listing gives, that is searched. + pub(super) max_file_size: u64, + /// The files considered between two pauses. + pub(super) slice_files: usize, + /// The bytes read between two pauses. + pub(super) slice_bytes: u64, + /// The most bytes of body in the answer. + pub(super) body_cap: usize, + /// How much of a long line is shown. + pub(super) window: Window, +} + +impl Default for Limits { + fn default() -> Limits { + Limits { + max_files: MAX_FILES, + max_bytes: MAX_BYTES, + max_file_size: MAX_FILE_BYTES, + slice_files: SLICE_FILES, + slice_bytes: SLICE_BYTES, + body_cap: BODY_CAP, + window: Window::default(), + } + } +} + +/// Why an entry was skipped, in the order the skip line names them. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum Reason { + /// A binary file. + Binary, + /// A directory whose listing failed. + UnreadableDirectory, + /// A file that could not be read. + UnreadableFile, + /// A file whose listed size is over the limit. + OverSize, + /// A symbolic link, which is never entered or read. + SymbolicLink, + /// An entry whose name has a control character. + ControlName, + /// An entry another call in this round holds a claim on. + Conflict, + /// An entry the run's policy refuses. + Denied, +} + +impl Reason { + /// Every reason, in the order the skip line names them. + pub(super) const ALL: [Reason; 8] = [ + Reason::Binary, + Reason::UnreadableDirectory, + Reason::UnreadableFile, + Reason::OverSize, + Reason::SymbolicLink, + Reason::ControlName, + Reason::Conflict, + Reason::Denied, + ]; +} + +/// The entries a search skipped, counted by reason. +#[derive(Debug, Default)] +pub(super) struct Skips { + /// How many entries were skipped for each reason in [`Reason::ALL`]. + counts: [usize; 8], + /// The first printable paths skipped, in walk order. + first: Vec, +} + +impl Skips { + /// Counts one skip for `reason`. `path` is the entry's path, or `None` + /// for a name that is never printed. + pub(super) fn record(&mut self, reason: Reason, path: Option<&str>) { + self.counts[reason as usize] += 1; + if let Some(path) = path + && self.first.len() < FIRST_KEPT + { + self.first.push(path.to_owned()); + } + } + + /// How many entries were skipped for `reason`. + pub(super) fn count(&self, reason: Reason) -> usize { + self.counts[reason as usize] + } + + /// How many entries were skipped for any reason. + pub(super) fn total(&self) -> usize { + self.counts.iter().sum() + } + + /// The first printable paths skipped, in walk order. + pub(super) fn first(&self) -> &[String] { + &self.first + } +} + +/// The files whose invalid UTF-8 was shown as U+FFFD. +#[derive(Debug, Default)] +pub(super) struct Replaced { + /// How many searched files needed a replacement. + pub(super) count: usize, + /// The first of their paths, in walk order. + pub(super) first: Vec, +} + +impl Replaced { + /// Notes that the file at `path` needed a replacement. + pub(super) fn record(&mut self, path: &str) { + self.count += 1; + if self.first.len() < FIRST_KEPT { + self.first.push(path.to_owned()); + } + } +} + +/// A limit that ended the search before the tree was done. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum Stop { + /// The search considered the most files it may. + Files, + /// The search read the most bytes it may. + Bytes, +} + +/// One line of a file's result: what is shown after its number. +#[derive(Debug)] +pub(super) struct Row { + /// The text shown, windowed or clipped when the line is long. + pub(super) text: String, + /// Whether the line is an entry. A context line is not. + pub(super) is_match: bool, +} + +/// What one file adds to the result. +#[derive(Debug)] +pub(super) struct FileHit { + /// The file's absolute virtual path. + pub(super) path: String, + /// The file's matching-line count, which `count` mode shows. + pub(super) count: usize, + /// The lines `content` mode shows, by line number from 1, in order. + pub(super) rows: BTreeMap, +} + +/// The outcome of a search that did not fail. +#[derive(Debug, Default)] +pub(super) struct Found { + /// The files that hold a kept entry, in walk order. + pub(super) hits: Vec, + /// The entries gathered after the offset: the ones kept, and the one + /// extra that only shows the head limit was reached. + pub(super) collected: usize, + /// Whether the extra entry was found. + pub(super) extra: bool, + /// Whether the search stopped collecting because the kept lines + /// already fill the body cap. + pub(super) capped: bool, + /// How many entries the search counted, the ones before the offset + /// included. + pub(super) seen: usize, + /// How many files were searched. + pub(super) searched: usize, + /// The entries that were skipped. + pub(super) skips: Skips, + /// The files that needed a replacement. + pub(super) replaced: Replaced, + /// The limit that ended the search early, when one did. + pub(super) stop: Option, +} diff --git a/crates/plugin-std/src/grep/matching.rs b/crates/plugin-std/src/grep/matching.rs new file mode 100644 index 000000000..2f7febbd3 --- /dev/null +++ b/crates/plugin-std/src/grep/matching.rs @@ -0,0 +1,84 @@ +//! Finding the matching lines of one decoded file. +//! +//! Outside `multiline` a line is a subject of its own. With it the whole +//! text is one subject, and every line a match touches is a matching line, +//! so a line number here is always the number Read prints for that line. + +use regex::Regex; + +/// Calls `emit` for each matching line of `text`, in line order and once +/// per line, until `emit` returns `false`. +/// +/// `lines` is `text` split the way Read splits it. `emit` gets the line's +/// index, from 0, and the byte offset in that line of the first match +/// that covers it: where the first match starts when it starts on this +/// line, and 0 for a line a match enters from an earlier one. +pub(super) fn each_match_line( + regex: &Regex, + multiline: bool, + text: &str, + lines: &[&str], + mut emit: impl FnMut(usize, usize) -> bool, +) { + if !multiline { + for (index, line) in lines.iter().enumerate() { + if let Some(found) = regex.find(line) + && !emit(index, found.start()) + { + return; + } + } + return; + } + let starts = line_starts(text); + // The first line no match has been reported for yet. + let mut next = 0; + for found in regex.find_iter(text) { + let Some(first) = line_of(&starts, text, found.start()) else { + continue; + }; + // An empty match touches only the line it starts on. + let last_byte = found.end().saturating_sub(1).max(found.start()); + let last = line_of(&starts, text, last_byte).unwrap_or(first); + let first_start = starts.get(first).copied().unwrap_or_default(); + for line in first.max(next)..=last { + let offset = if line == first { + found.start() - first_start + } else { + 0 + }; + let width = lines.get(line).map_or(0, |line| line.len()); + if !emit(line, offset.min(width)) { + return; + } + } + next = next.max(last + 1); + } +} + +/// The byte offset where each line of `text` starts, split the way +/// `split_lines` splits: on `\n`, with no line after a final newline. +fn line_starts(text: &str) -> Vec { + let mut starts = Vec::new(); + let mut start = 0; + for piece in text.split_terminator('\n') { + starts.push(start); + start += piece.len() + 1; + } + starts +} + +/// The index of the line that holds byte `offset` of `text`, or `None` +/// when no line does: the text is empty, or `offset` is the end of a text +/// that ends in a newline. +fn line_of(starts: &[usize], text: &str, offset: usize) -> Option { + let after = starts.partition_point(|start| *start <= offset); + let line = after.checked_sub(1)?; + if offset == text.len() && text.ends_with('\n') { + return None; + } + Some(line) +} + +#[cfg(test)] +mod tests; diff --git a/crates/plugin-std/src/grep/matching/tests.rs b/crates/plugin-std/src/grep/matching/tests.rs new file mode 100644 index 000000000..db9a88532 --- /dev/null +++ b/crates/plugin-std/src/grep/matching/tests.rs @@ -0,0 +1,95 @@ +//! Unit tests for the matching-line stream. + +use regex::RegexBuilder; + +use super::{each_match_line, line_of, line_starts}; +use crate::text::split_lines; + +/// The matching lines of `text` for `pattern`, each with its first-match +/// offset, using the flags Grep compiles with. +fn matches(pattern: &str, multiline: bool, text: &str) -> Vec<(usize, usize)> { + let regex = RegexBuilder::new(pattern) + .multi_line(true) + .dot_matches_new_line(multiline) + .crlf(multiline) + .build() + .expect("the pattern compiles"); + let lines = split_lines(text); + let mut found = Vec::new(); + each_match_line(®ex, multiline, text, &lines, |line, offset| { + found.push((line, offset)); + true + }); + found +} + +#[test] +fn line_by_line_reports_each_matching_line_once_with_its_first_match() { + assert_eq!( + matches("a", false, "xa\nb\nana\n"), + [(0, 1), (2, 0)], + "the second match on a line does not repeat the line" + ); +} + +#[test] +fn a_multiline_match_touches_every_line_from_its_first_byte_to_its_last() { + assert_eq!( + matches("b\nc", true, "a\nb\nc\nd\n"), + [(1, 0), (2, 0)], + "lines 2 and 3" + ); + assert_eq!( + matches("b\n", true, "a\nb\nc\n"), + [(1, 0)], + "a match that ends with the newline touches only the line it ends" + ); +} + +#[test] +fn a_line_two_matches_touch_is_reported_once_with_the_first_matchs_offset() { + assert_eq!( + matches("a\nb|c", true, "xa\nbc\n"), + [(0, 1), (1, 0)], + "line 2 holds the end of one match and the whole of another" + ); +} + +#[test] +fn an_empty_match_touches_the_line_it_starts_on_and_none_after_a_final_newline() { + assert_eq!(matches("$", true, "a\nb\n"), [(0, 1), (1, 1)]); + assert_eq!(matches("$", true, "a"), [(0, 1)]); + assert_eq!(matches("^", true, ""), []); +} + +#[test] +fn the_stream_stops_when_the_callback_says_so() { + let regex = RegexBuilder::new("a").build().expect("compiles"); + let text = "a\na\na\n"; + let lines = split_lines(text); + let mut seen = 0; + each_match_line(®ex, false, text, &lines, |_, _| { + seen += 1; + seen < 2 + }); + assert_eq!(seen, 2); +} + +#[test] +fn line_starts_agree_with_split_lines_for_every_shape_of_ending() { + for text in ["", "a", "a\n", "a\n\n", "\n", "a\r\nb\r\n", "a\n\nb"] { + assert_eq!(line_starts(text).len(), split_lines(text).len(), "{text:?}"); + } +} + +#[test] +fn line_of_places_offsets_on_the_line_that_holds_them() { + let text = "a\n\nb"; + let starts = line_starts(text); + assert_eq!(starts, [0, 2, 3]); + assert_eq!(line_of(&starts, text, 0), Some(0)); + assert_eq!(line_of(&starts, text, 1), Some(0)); + assert_eq!(line_of(&starts, text, 2), Some(1)); + assert_eq!(line_of(&starts, text, 4), Some(2)); + assert_eq!(line_of(&line_starts("a\n"), "a\n", 2), None); +} diff --git a/crates/plugin-std/src/grep/pause.rs b/crates/plugin-std/src/grep/pause.rs new file mode 100644 index 000000000..06741b5fa --- /dev/null +++ b/crates/plugin-std/src/grep/pause.rs @@ -0,0 +1,32 @@ +//! The pause that lets a running search be cancelled. +//! +//! The Harness polls a call's future inside its effect loop, and a cancel +//! or a stop drops that future. A future that never returns `Pending` +//! cannot be dropped until it finishes, so Grep awaits a [`Pause`] every +//! so often. The pause wakes its own task and returns `Pending` once, +//! which hands control back to the Host's executor, and returns `Ready` +//! on the next poll. It names no async runtime. + +use std::future::Future; +use std::pin::Pin; +use std::task::{Context, Poll}; + +/// A future that is `Pending` for one poll and then `Ready`. +#[derive(Debug, Default)] +pub(super) struct Pause { + /// Whether the one `Pending` has been returned. + yielded: bool, +} + +impl Future for Pause { + type Output = (); + + fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<()> { + if self.yielded { + return Poll::Ready(()); + } + self.yielded = true; + cx.waker().wake_by_ref(); + Poll::Pending + } +} diff --git a/crates/plugin-std/src/grep/render.rs b/crates/plugin-std/src/grep/render.rs new file mode 100644 index 000000000..dfa13fc35 --- /dev/null +++ b/crates/plugin-std/src/grep/render.rs @@ -0,0 +1,380 @@ +//! The render: turns what a search found into the text Grep answers with. +//! +//! There is no envelope. The answer is the body, and then, when there is +//! anything to add, a blank line and the extra lines: the replacement +//! note, the skip line, and the paging or limit notice. The extra lines +//! never count against the body cap. + +use runvil_plugin::ToolOutput; + +use super::args::{Mode, Request}; +use super::found::{Found, Limits, Reason, Stop}; +use crate::check::plural; + +/// The line printed between two groups of one file that are not next to +/// each other, when context is on. +const GAP: &str = " --"; + +/// What a gap line costs against the cap, with its newline. +pub(super) const GAP_COST: usize = GAP.len() + 1; + +/// The line that tells a model nothing was filtered by ignore rules. +const IGNORE_NOTE: &str = + "[No ignore rules ran: .gitignore is not applied, and only entries named .git are skipped.]"; + +/// The line that says what `{`, `[`, and `?` do in a glob. +const GLOB_HINT: &str = "{, [ and ? in glob match themselves; there is no alternation or \ + character class. Call Grep once per alternative, or filter with type."; + +/// What the body cap costs a line: its text and its newline. +fn line_cost(text_len: usize) -> usize { + text_len + 1 +} + +/// What a `content` line costs against the cap: two spaces, the number, +/// the `:` or `-`, the text, and the newline. +pub(super) fn row_cost(number: usize, text: &str) -> usize { + line_cost(2 + number.to_string().len() + 1 + text.len()) +} + +/// What a `files_with_matches` or `count` line costs against the cap. +pub(super) fn summary_cost(mode: Mode, path: &str, count: usize) -> usize { + match mode { + Mode::Count => line_cost(path.len() + 1 + count.to_string().len()), + _ => line_cost(path.len()), + } +} + +/// What kind of line a row is. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Kind { + /// A file's path, which opens its group in `content` mode. + Path, + /// The empty line between two files. + Blank, + /// The gap line. + Gap, + /// A matching line. + Match, + /// A context line. + Context, + /// A path, or a path and its count, in the other two modes. + Summary, +} + +/// One line of the body. +#[derive(Debug)] +struct Out { + /// The line, without its newline. + text: String, + /// What the line is. + kind: Kind, + /// The line number of a matching or context line, and 0 for others. + line: usize, +} + +/// The body: the lines that fit under the cap. +#[derive(Debug)] +struct Body { + /// The lines joined with `\n`. + text: String, + /// How many entries the lines show. + shown: usize, + /// Whether the cap cut the body short. + cut: bool, +} + +/// The answer to a search that found `found`. +pub(super) fn render(found: &Found, request: &Request<'_>, limits: &Limits) -> ToolOutput { + let note = replacement_note(found); + let skip = skip_line(found, limits); + let stopped = found.stop.map(|stop| limit_notice(stop, limits)); + if found.seen == 0 { + let text = format!( + "No matches found. Searched {}.", + plural(found.searched, "file") + ); + let trusted = note.is_none() && skip.is_none(); + let mut extras: Vec = [note, skip].into_iter().flatten().collect(); + extras.push(IGNORE_NOTE.to_owned()); + if request + .glob + .as_ref() + .is_some_and(|(glob, _)| glob.contains(['{', '[', '?'])) + { + extras.push(GLOB_HINT.to_owned()); + } + extras.extend(stopped); + let text = assemble(text, &extras); + return if trusted { + ToolOutput::trusted(text) + } else { + ToolOutput::untrusted(text) + }; + } + if found.collected == 0 { + let extras: Vec = stopped.into_iter().collect(); + let text = format!("No more results at offset={}.", request.offset); + return ToolOutput::untrusted(assemble(text, &extras)); + } + let body = body(found, request, limits.body_cap); + let capped = body.cut || found.capped; + let text = if body.shown == 0 { + format!( + "[Output stopped at {} bytes before the first result fit. Narrow the search with \ + path, glob or type.]", + limits.body_cap + ) + } else { + body.text + }; + let tail = if body.shown == 0 { + stopped + } else { + stopped.or_else(|| paging_notice(found, request, body.shown, capped, limits)) + }; + let extras: Vec = [note, skip, tail].into_iter().flatten().collect(); + ToolOutput::untrusted(assemble(text, &extras)) +} + +/// `body`, then a blank line and the `extras` on their own lines, when +/// there are any. +fn assemble(mut body: String, extras: &[String]) -> String { + if !extras.is_empty() { + body.push_str("\n\n"); + body.push_str(&extras.join("\n")); + } + body +} + +/// The body for `found`, with at most `cap` bytes of lines. +fn body(found: &Found, request: &Request<'_>, cap: usize) -> Body { + let rows = match request.mode { + Mode::Content => content_rows(found, request.before > 0 || request.after > 0), + Mode::FilesWithMatches => found + .hits + .iter() + .map(|hit| Out { + text: hit.path.clone(), + kind: Kind::Summary, + line: 0, + }) + .collect(), + Mode::Count => found + .hits + .iter() + .map(|hit| Out { + text: format!("{}:{}", hit.path, hit.count), + kind: Kind::Summary, + line: 0, + }) + .collect(), + }; + cut(rows, cap, request) +} + +/// The `content` lines of every file that kept one: its path, then its +/// lines in order, with a gap line between two groups that are not next +/// to each other when `gaps` is on, and a blank line between files. +fn content_rows(found: &Found, gaps: bool) -> Vec { + let mut rows = Vec::new(); + for hit in found.hits.iter().filter(|hit| !hit.rows.is_empty()) { + if !rows.is_empty() { + rows.push(Out { + text: String::new(), + kind: Kind::Blank, + line: 0, + }); + } + rows.push(Out { + text: hit.path.clone(), + kind: Kind::Path, + line: 0, + }); + let mut previous: Option = None; + for (number, row) in &hit.rows { + if gaps && previous.is_some_and(|previous| *number > previous + 1) { + rows.push(Out { + text: GAP.to_owned(), + kind: Kind::Gap, + line: 0, + }); + } + previous = Some(*number); + let mark = if row.is_match { ':' } else { '-' }; + rows.push(Out { + text: format!(" {number}{mark}{}", row.text), + kind: if row.is_match { + Kind::Match + } else { + Kind::Context + }, + line: *number, + }); + } + } + rows +} + +/// `rows` cut before the line that would pass `cap`. +/// +/// A cut that falls inside a file's lines leaves nothing dangling: the +/// body ends at the last matching line it shows, or at the last of that +/// line's own trailing context lines, never at a path, a gap, or the +/// leading context of a match that did not fit. +fn cut(mut rows: Vec, cap: usize, request: &Request<'_>) -> Body { + let mut used = 0; + let mut keep = rows.len(); + for (index, row) in rows.iter().enumerate() { + let cost = line_cost(row.text.len()); + if used + cost > cap { + keep = index; + break; + } + used += cost; + } + let cut = keep < rows.len(); + rows.truncate(keep); + if cut && request.mode == Mode::Content { + let end = rows + .iter() + .rposition(|row| row.kind == Kind::Match) + .map_or(0, |last| { + let limit = rows[last].line.saturating_add(request.after); + let trailing = rows[last + 1..] + .iter() + .take_while(|row| row.kind == Kind::Context && row.line <= limit) + .count(); + last + 1 + trailing + }); + rows.truncate(end); + } + let shown = rows + .iter() + .filter(|row| matches!(row.kind, Kind::Match | Kind::Summary)) + .count(); + let text = rows + .iter() + .map(|row| row.text.as_str()) + .collect::>() + .join("\n"); + Body { text, shown, cut } +} + +/// The paging notice, or `None` when there is nothing to say. +/// +/// There is no cache, so each call runs from the start. A cut says where +/// to continue. A page after the first that reaches the end gives an +/// exact total. A first page that nothing cut gets no notice. +fn paging_notice( + found: &Found, + request: &Request<'_>, + shown: usize, + capped: bool, + limits: &Limits, +) -> Option { + let offset = request.offset; + let unit = if request.mode == Mode::Content { + "matching lines" + } else { + "files" + }; + let cut = capped || found.extra; + if !cut && offset == 0 { + return None; + } + let first = offset + 1; + let last = offset + shown; + let total = offset + found.collected; + if !cut { + return Some(format!("[Showing {first}-{last} of {total} {unit}.]")); + } + let stopped = if capped { + format!(" Output stopped at {} bytes.", limits.body_cap) + } else { + String::new() + }; + Some(format!( + "[Showing {first}-{last} of at least {total} {unit}.{stopped} Use offset={last} to see \ + more, or narrow the search.]" + )) +} + +/// The note that names the files whose invalid UTF-8 was replaced. +fn replacement_note(found: &Found) -> Option { + let replaced = &found.replaced; + if replaced.count == 0 { + return None; + } + let mut paths = replaced.first.join(", "); + if replaced.count > replaced.first.len() { + paths = format!( + "{paths}, and {} more", + replaced.count - replaced.first.len() + ); + } + Some(format!( + "[Note: invalid UTF-8 in {} was shown as U+FFFD: {paths}]", + plural(replaced.count, "file") + )) +} + +/// The line that counts the skipped entries by reason and names the first +/// few of them. +fn skip_line(found: &Found, limits: &Limits) -> Option { + let skips = &found.skips; + let total = skips.total(); + if total == 0 { + return None; + } + let phrases: Vec = Reason::ALL + .iter() + .filter_map(|reason| { + let n = skips.count(*reason); + (n > 0).then(|| phrase(*reason, n, limits)) + }) + .collect(); + let first = if skips.first().is_empty() { + String::new() + } else { + format!(" First: {}", skips.first().join(", ")) + }; + Some(format!( + "[Skipped {}: {}.{first}]", + plural(total, "entry"), + phrases.join(", ") + )) +} + +/// The phrase that says `n` entries were skipped for `reason`. +fn phrase(reason: Reason, n: usize, limits: &Limits) -> String { + match reason { + Reason::Binary => plural(n, "binary file"), + Reason::UnreadableDirectory => { + format!("{} that could not be read", plural(n, "directory")) + } + Reason::UnreadableFile => format!("{} that could not be read", plural(n, "file")), + Reason::OverSize => format!("{} over {} bytes", plural(n, "file"), limits.max_file_size), + Reason::SymbolicLink => plural(n, "symbolic link"), + Reason::ControlName => format!("{} with a control character", plural(n, "name")), + Reason::Conflict => format!("{} held by another call in this round", plural(n, "entry")), + Reason::Denied => format!("{} denied by policy", plural(n, "entry")), + } +} + +/// The notice for a search a limit ended early. It offers no `offset`, +/// because a re-run stops at the same place. +fn limit_notice(stop: Stop, limits: &Limits) -> String { + match stop { + Stop::Files => format!( + "[Search stopped after considering {} files. Narrow path, glob or type to search the \ + rest.]", + limits.max_files + ), + Stop::Bytes => format!( + "[Search stopped after reading {} bytes. Narrow path, glob or type to search the \ + rest.]", + limits.max_bytes + ), + } +} diff --git a/crates/plugin-std/src/grep/source.rs b/crates/plugin-std/src/grep/source.rs new file mode 100644 index 000000000..7cfd6c57f --- /dev/null +++ b/crates/plugin-std/src/grep/source.rs @@ -0,0 +1,43 @@ +//! The file operations Grep needs, behind a trait. +//! +//! The call's [`Access`] implements it, so every operation Grep performs +//! goes through the run's mode and policy. The seam exists so that a test +//! can supply what the in-memory backend cannot produce: a symbolic link, +//! a name with a control character, an over-size entry, a denied read, and +//! a claim conflict. + +use runvil_plugin::{Access, Entry, Stat, VfsError}; + +/// Where Grep reads the tree from. +pub(super) trait Source { + /// Metadata for `path`, without following a link. + fn stat(&self, path: &str) -> Result; + + /// The entries of the directory at `path`, in no promised order. + fn list(&self, path: &str) -> Result, VfsError>; + + /// The whole contents of the file at `path`. + fn read(&self, path: &str) -> Result, VfsError>; + + /// At most the first `len` bytes of the file at `path`, for a path + /// whose size is not known beforehand. + fn read_prefix(&self, path: &str, len: u64) -> Result, VfsError>; +} + +impl Source for Access { + fn stat(&self, path: &str) -> Result { + Access::stat(self, path) + } + + fn list(&self, path: &str) -> Result, VfsError> { + Access::list(self, path) + } + + fn read(&self, path: &str) -> Result, VfsError> { + Access::read(self, path) + } + + fn read_prefix(&self, path: &str, len: u64) -> Result, VfsError> { + Access::read_range(self, path, 0, len) + } +} diff --git a/crates/plugin-std/src/grep/tests.rs b/crates/plugin-std/src/grep/tests.rs new file mode 100644 index 000000000..c8293c7d1 --- /dev/null +++ b/crates/plugin-std/src/grep/tests.rs @@ -0,0 +1,116 @@ +//! Unit tests for the `Grep` tool. This file holds the helpers the topic +//! files share: a call over a fresh in-memory filesystem, a seeded tree, +//! and the ways to drive Grep, through its arguments and through lowered +//! limits over any [`Source`]. + +mod arguments; +mod fake; +mod filters; +mod paging; +mod patterns; +mod render; +mod search; +mod skips; +mod text; +mod window; + +use std::future::Future; +use std::pin::pin; +use std::task::{Context, Poll, Waker}; + +use runvil_plugin::testing::TestCall; +use runvil_plugin::{OutputTrust, ToolError, ToolId, ToolOutput}; +use serde_json::Value; + +use super::args; +use super::found::Limits; +use super::source::Source; +use crate::spec::{self, Definition}; + +/// Drives a future to its output, polling again after each `Pending`. +fn block_on(future: F) -> F::Output { + let mut future = pin!(future); + let mut cx = Context::from_waker(Waker::noop()); + loop { + if let Poll::Ready(output) = future.as_mut().poll(&mut cx) { + return output; + } + } +} + +/// Drives a future to its output and counts how many times it returned +/// `Pending` on the way. +fn count_pendings(future: F) -> (F::Output, usize) { + let mut future = pin!(future); + let mut cx = Context::from_waker(Waker::noop()); + let mut pending = 0; + loop { + if let Poll::Ready(output) = future.as_mut().poll(&mut cx) { + return (output, pending); + } + pending += 1; + } +} + +/// A call of `std/grep` over a fresh in-memory filesystem. +fn new_call() -> TestCall { + TestCall::new(ToolId::parse("std/grep").expect("a two-segment tool id parses")) +} + +/// A call with one file at each of `files`, a path beside its text, which +/// makes every directory on the way to it. +fn seeded(files: &[(&str, &str)]) -> TestCall { + let call = new_call(); + for (path, text) in files { + call.context() + .access() + .write(path, text.as_bytes()) + .expect("a seeded file is written"); + } + call +} + +/// The `grep` definition the Pack builds the tool from. +fn definition() -> &'static Definition { + spec::all() + .expect("every embedded definition parses") + .iter() + .find(|definition| definition.segment == "grep") + .expect("the grep definition exists") +} + +/// Calls Grep with `args` as the Pack does: the schema check first, then +/// the tool. +fn grep(call: &TestCall, args: &Value) -> Result { + let definition = definition(); + crate::check::check(&definition.wire_name, &definition.parameters, args)?; + block_on(super::call(call.context().access(), definition, args)) +} + +/// Calls Grep over `source` with `args`, under `limits`. +fn grep_over(source: &impl Source, args: &Value, limits: &Limits) -> Result { + let definition = definition(); + crate::check::check(&definition.wire_name, &definition.parameters, args)?; + let request = args::parse(definition, args)?; + block_on(super::search(source, "Grep", &request, limits)) +} + +/// Calls Grep over the call's filesystem under `limits`. +fn grep_limited(call: &TestCall, args: &Value, limits: &Limits) -> Result { + grep_over(call.context().access(), args, limits) +} + +/// The text of a successful search, which must be untrusted output. +fn text_of(result: Result) -> String { + let output = result.expect("the search succeeds"); + assert_eq!(output.trust(), OutputTrust::Untrusted, "{}", output.text()); + output.text().to_owned() +} + +/// The message of a failed search. +fn failure_of(result: Result) -> String { + match result { + Ok(output) => panic!("the search fails, but it answered {:?}", output.text()), + Err(error) => error.to_string(), + } +} diff --git a/crates/plugin-std/src/grep/tests/arguments.rs b/crates/plugin-std/src/grep/tests/arguments.rs new file mode 100644 index 000000000..d427a54c2 --- /dev/null +++ b/crates/plugin-std/src/grep/tests/arguments.rs @@ -0,0 +1,246 @@ +//! What Grep refuses about its arguments, and in which order. + +use runvil_plugin::ToolErrorKind; +use serde_json::{Value, json}; + +use super::{definition, failure_of, grep, new_call, seeded}; +use crate::grep::args::{self, Mode}; + +/// The thirteen parameters, sorted bytewise, as an undeclared-key refusal +/// lists them. +const PARAMETERS: &str = "`-A`, `-B`, `-C`, `-i`, `glob`, `head_limit`, `literal`, `multiline`, \ + `offset`, `output_mode`, `path`, `pattern` and `type`"; + +/// The refusal text for a call with `args`. +fn refusal(args: &Value) -> String { + failure_of(grep(&new_call(), args)) +} + +/// The request `args` parse to. +fn parsed(args: &Value) -> args::Request<'_> { + args::parse(definition(), args).expect("the arguments parse") +} + +#[test] +fn a_missing_pattern_names_the_parameter() { + let error = grep(&new_call(), &json!({})).expect_err("pattern is required"); + assert_eq!( + error.to_string(), + "std_Grep: `pattern` is required, and the call left it out" + ); + assert_eq!(error.kind(), ToolErrorKind::InvalidArguments); +} + +#[test] +fn an_undeclared_key_is_refused_naming_it_and_listing_the_sorted_parameters() { + let error = grep(&new_call(), &json!({ "pattern": "x", "context": 3 })) + .expect_err("context is not a parameter"); + assert_eq!( + error.to_string(), + format!( + "std_Grep: `context` is not a parameter of this tool. The parameters are {PARAMETERS}." + ) + ); + assert_eq!(error.kind(), ToolErrorKind::InvalidArguments); +} + +#[test] +fn the_one_undeclared_key_that_is_accepted_is_n_when_it_is_true() { + let call = seeded(&[("/a.txt", "needle\n")]); + for accepted in [json!(true), Value::Null] { + let output = grep(&call, &json!({ "pattern": "needle", "-n": accepted })) + .expect("-n true or null is accepted"); + assert_eq!(output.text(), "/a.txt\n 1:needle"); + } + let off = "std_Grep: `-n` must be true or left out, because line numbers are always shown."; + for refused in [json!(false), json!("yes"), json!(1)] { + assert_eq!(refusal(&json!({ "pattern": "x", "-n": refused })), off); + } +} + +#[test] +fn an_empty_pattern_is_refused_with_the_advice_to_pass_a_dot() { + assert_eq!( + refusal(&json!({ "pattern": "" })), + "std_Grep: `pattern` is empty. Pass \".\" to match every line." + ); +} + +#[test] +fn a_negative_context_is_refused_naming_the_key_and_the_value_in_a_fixed_order() { + assert_eq!( + refusal(&json!({ "pattern": "x", "-A": -1 })), + "std_Grep: `-A` is -1. Pass 0 or more context lines." + ); + assert_eq!( + refusal(&json!({ "pattern": "x", "-C": -2, "-B": -3, "-A": -4 })), + "std_Grep: `-A` is -4. Pass 0 or more context lines.", + "-A comes first" + ); + assert_eq!( + refusal(&json!({ "pattern": "x", "-C": -2, "-B": -3 })), + "std_Grep: `-B` is -3. Pass 0 or more context lines.", + "-B comes before -C" + ); + assert_eq!( + refusal(&json!({ "pattern": "x", "-C": -1.5 })), + "std_Grep: `-C` is -1.5. Pass 0 or more context lines." + ); +} + +#[test] +fn a_negative_head_limit_or_offset_is_refused_head_limit_first() { + assert_eq!( + refusal(&json!({ "pattern": "x", "head_limit": -5 })), + "std_Grep: `head_limit` is -5. Pass 0 or more." + ); + assert_eq!( + refusal(&json!({ "pattern": "x", "offset": -1, "head_limit": -5 })), + "std_Grep: `head_limit` is -5. Pass 0 or more." + ); + assert_eq!( + refusal(&json!({ "pattern": "x", "offset": -1 })), + "std_Grep: `offset` is -1. Pass 0 or more." + ); +} + +#[test] +fn an_unknown_type_is_refused_with_the_types_to_try() { + assert_eq!( + refusal(&json!({ "pattern": "x", "type": "foo" })), + "std_Grep: Unrecognized file type \"foo\". Use a type such as js, ts, py, rust, go or \ + java, or filter with glob." + ); +} + +#[test] +fn an_invalid_glob_is_refused_with_the_rule_it_broke() { + assert_eq!( + refusal(&json!({ "pattern": "x", "glob": "a***" })), + "std_Grep: `glob` is invalid: \"a***\" does not parse: more than two consecutive '*' \ + are not supported." + ); +} + +#[test] +fn a_whitespace_only_path_is_refused_echoing_it() { + assert_eq!( + refusal(&json!({ "pattern": "x", "path": " " })), + "std_Grep: `path` is empty or only whitespace, and the call passed \" \"" + ); +} + +#[test] +fn a_call_that_breaks_several_rules_always_hears_about_the_first_in_order() { + let mut args = json!({ + "pattern": "", + "context": 1, + "-n": false, + "-A": -1, + "head_limit": -1, + "type": "foo", + "glob": "a***", + "path": " ", + }); + let order = [ + ("context", "`context` is not a parameter"), + ("-n", "`-n` must be true"), + ("pattern", "`pattern` is empty"), + ("-A", "`-A` is -1"), + ("head_limit", "`head_limit` is -1"), + ("type", "Unrecognized file type"), + ("glob", "`glob` is invalid"), + ("path", "`path` is empty or only whitespace"), + ]; + for (key, expected) in order { + let message = refusal(&args); + assert!( + message.contains(expected), + "expected {expected:?} before {key} is fixed: {message}" + ); + match key { + "pattern" => args["pattern"] = json!("("), + other => { + args.as_object_mut().expect("an object").remove(other); + } + } + } + assert!( + refusal(&args).contains("Invalid regular expression \"(\""), + "the pattern is compiled after every other check" + ); +} + +#[test] +fn an_empty_path_glob_or_type_reads_as_absent() { + let call = seeded(&[("/src/a.rs", "needle\n"), ("/b.txt", "needle\n")]); + let output = grep( + &call, + &json!({ + "pattern": "needle", + "output_mode": "files_with_matches", + "path": "", + "glob": "", + "type": "", + }), + ) + .expect("empty strings read as absent"); + assert_eq!(output.text(), "/b.txt\n/src/a.rs"); +} + +#[test] +fn head_limit_and_context_round_up_and_offset_rounds_down() { + let args = json!({ "pattern": "x", "head_limit": 2.2, "offset": 2.9, "-A": 0.2 }); + let request = parsed(&args); + assert_eq!(request.limit, Some(3)); + assert_eq!(request.offset, 2); + assert_eq!(request.after, 1); +} + +#[test] +fn head_limit_zero_and_an_absent_head_limit_both_mean_no_limit() { + for args in [ + json!({ "pattern": "x", "head_limit": 0 }), + json!({ "pattern": "x" }), + json!({ "pattern": "x", "head_limit": null }), + ] { + assert_eq!(parsed(&args).limit, None, "{args}"); + } +} + +#[test] +fn context_c_sets_both_sides_and_a_and_b_override_it_even_at_zero() { + let sides = |args: Value| { + let request = parsed(&args); + (request.before, request.after) + }; + assert_eq!(sides(json!({ "pattern": "x", "-C": 2 })), (2, 2)); + assert_eq!(sides(json!({ "pattern": "x", "-C": 2, "-A": 0 })), (2, 0)); + assert_eq!(sides(json!({ "pattern": "x", "-C": 2, "-B": 5 })), (5, 2)); + assert_eq!( + sides(json!({ "pattern": "x", "-C": 2, "-B": 0, "-A": 1 })), + (0, 1) + ); + assert_eq!(sides(json!({ "pattern": "x", "-A": 3 })), (0, 3)); +} + +#[test] +fn context_is_ignored_outside_content_mode() { + for mode in ["files_with_matches", "count"] { + let args = json!({ "pattern": "x", "output_mode": mode, "-C": 4 }); + let request = parsed(&args); + assert_eq!((request.before, request.after), (0, 0), "{mode}"); + } + let args = json!({ "pattern": "x", "-C": 4 }); + let request = parsed(&args); + assert_eq!(request.mode, Mode::Content, "content is the default mode"); + assert_eq!((request.before, request.after), (4, 4)); +} + +#[test] +fn a_value_past_the_range_of_usize_saturates_instead_of_failing() { + let args = json!({ "pattern": "x", "offset": 1e30, "-C": 1e30 }); + let request = parsed(&args); + assert_eq!(request.offset, usize::MAX); + assert_eq!(request.before, usize::MAX); +} diff --git a/crates/plugin-std/src/grep/tests/fake.rs b/crates/plugin-std/src/grep/tests/fake.rs new file mode 100644 index 000000000..60e805121 --- /dev/null +++ b/crates/plugin-std/src/grep/tests/fake.rs @@ -0,0 +1,129 @@ +//! A [`Source`] the tests build by hand, for what the in-memory backend +//! cannot produce: a link, a control-character name, an over-size entry, +//! a denied read, a claim conflict, and an I/O failure. + +use std::collections::BTreeMap; + +use runvil_plugin::{Entry, FileType, Stat, VfsError}; + +use crate::grep::source::Source; + +/// One node of the fake tree. +#[derive(Debug, Clone)] +pub(super) enum Node { + /// A file with these bytes. + File(Vec), + /// A symbolic link to a directory. + Link, + /// A symbolic link to a file with these bytes. + LinkToFile(Vec), + /// A pipe. + Pipe, + /// A file whose listing says it holds this many bytes, and whose + /// bytes are the second field. + Sized(u64, Vec), + /// A directory whose listing fails with this error. + ListFails(VfsError), + /// A file whose read fails with this error. + ReadFails(VfsError), +} + +/// A tree of nodes by absolute path, with `/` always a directory. +#[derive(Debug, Default)] +pub(super) struct FakeSource { + /// Every node but the root. + nodes: BTreeMap, +} + +impl FakeSource { + /// The tree with `node` added at `path`. + pub(super) fn with(mut self, path: &str, node: Node) -> FakeSource { + self.nodes.insert(path.to_owned(), node); + self + } + + /// The tree with the text file `text` at `path`. + pub(super) fn file(self, path: &str, text: &str) -> FakeSource { + self.with(path, Node::File(text.as_bytes().to_vec())) + } + + /// The metadata of `node`. + fn stat_of(node: &Node) -> Stat { + match node { + Node::File(bytes) => Stat::new(FileType::File, bytes.len() as u64), + Node::Sized(size, _) => Stat::new(FileType::File, *size), + Node::ReadFails(_) => Stat::new(FileType::File, 1), + Node::ListFails(_) => Stat::new(FileType::Directory, 0), + Node::Link | Node::LinkToFile(_) => Stat::new(FileType::Symlink, 0), + Node::Pipe => Stat::new(FileType::Fifo, 0), + } + } + + /// The error for a path that is not there. + fn missing(path: &str) -> VfsError { + VfsError::NotFound { + path: path.to_owned(), + } + } +} + +/// The directory that holds `path`, and the name it has there. +fn split(path: &str) -> (&str, &str) { + match path.rsplit_once('/') { + Some(("", name)) => ("/", name), + Some((parent, name)) => (parent, name), + None => ("/", path), + } +} + +impl Source for FakeSource { + fn stat(&self, path: &str) -> Result { + if path == "/" { + return Ok(Stat::new(FileType::Directory, 0)); + } + self.nodes + .get(path) + .map(Self::stat_of) + .ok_or_else(|| Self::missing(path)) + } + + fn list(&self, path: &str) -> Result, VfsError> { + if path != "/" { + match self.nodes.get(path) { + None => return Err(Self::missing(path)), + Some(Node::ListFails(error)) => return Err(error.clone()), + Some(Node::Link) => {} + Some(_) => { + return Err(VfsError::NotADirectory { + path: path.to_owned(), + }); + } + } + } + Ok(self + .nodes + .iter() + .filter(|(child, _)| split(child).0 == path) + .map(|(child, node)| Entry::new(split(child).1, Self::stat_of(node))) + .collect()) + } + + fn read(&self, path: &str) -> Result, VfsError> { + match self.nodes.get(path) { + Some(Node::File(bytes) | Node::Sized(_, bytes) | Node::LinkToFile(bytes)) => { + Ok(bytes.clone()) + } + Some(Node::ReadFails(error)) => Err(error.clone()), + Some(Node::Link | Node::ListFails(_)) => Err(VfsError::IsADirectory { + path: path.to_owned(), + }), + Some(Node::Pipe) | None => Err(Self::missing(path)), + } + } + + fn read_prefix(&self, path: &str, len: u64) -> Result, VfsError> { + let mut bytes = self.read(path)?; + bytes.truncate(usize::try_from(len).unwrap_or(usize::MAX)); + Ok(bytes) + } +} diff --git a/crates/plugin-std/src/grep/tests/filters.rs b/crates/plugin-std/src/grep/tests/filters.rs new file mode 100644 index 000000000..1d9a35310 --- /dev/null +++ b/crates/plugin-std/src/grep/tests/filters.rs @@ -0,0 +1,125 @@ +//! The `glob` and `type` filters, and the one case where neither applies. + +use serde_json::json; + +use super::{grep, seeded, text_of}; + +/// The paths of the files with a match for `x` under `args`. +fn files(call: &runvil_plugin::testing::TestCall, mut args: serde_json::Value) -> String { + args["pattern"] = json!("x"); + args["output_mode"] = json!("files_with_matches"); + text_of(grep(call, &args)) +} + +#[test] +fn a_glob_without_a_leading_slash_matches_at_any_depth() { + let call = seeded(&[("/a.js", "x\n"), ("/d/e/b.js", "x\n"), ("/c.ts", "x\n")]); + assert_eq!(files(&call, json!({ "glob": "*.js" })), "/a.js\n/d/e/b.js"); +} + +#[test] +fn a_glob_with_a_leading_slash_is_a_full_path() { + let call = seeded(&[ + ("/src/a.rs", "x\n"), + ("/src/deep/b.rs", "x\n"), + ("/other/c.rs", "x\n"), + ("/src/d.txt", "x\n"), + ]); + assert_eq!( + files(&call, json!({ "glob": "/src/**/*.rs" })), + "/src/a.rs\n/src/deep/b.rs" + ); +} + +#[test] +fn a_type_keeps_the_files_with_its_extensions() { + let call = seeded(&[ + ("/lib.rs", "x\n"), + ("/main.RS", "x\n"), + ("/notes.md", "x\n"), + ("/sub/mod.rs", "x\n"), + ]); + assert_eq!( + files(&call, json!({ "type": "rust" })), + "/lib.rs\n/sub/mod.rs" + ); +} + +#[test] +fn a_glob_and_a_type_together_keep_only_what_passes_both() { + let call = seeded(&[("/a/x.rs", "x\n"), ("/b/x.rs", "x\n"), ("/a/y.txt", "x\n")]); + assert_eq!( + files(&call, json!({ "glob": "/a/**", "type": "rust" })), + "/a/x.rs" + ); +} + +#[test] +fn a_named_single_file_ignores_both_filters() { + let call = seeded(&[("/c.ts", "x\n")]); + let output = files( + &call, + json!({ "path": "/c.ts", "glob": "*.js", "type": "rust" }), + ); + assert_eq!(output, "/c.ts"); +} + +#[test] +fn a_file_a_filter_drops_is_neither_searched_nor_a_skip() { + let call = seeded(&[("/a.js", "x\n"), ("/b.bin.ts", "x\n")]); + let output = + grep(&call, &json!({ "pattern": "zzz", "glob": "*.js" })).expect("the search succeeds"); + assert!( + output + .text() + .starts_with("No matches found. Searched 1 file."), + "{}", + output.text() + ); + assert!(!output.text().contains("Skipped"), "{}", output.text()); +} + +#[test] +fn a_zero_match_search_with_a_brace_glob_adds_the_hint_line() { + let call = seeded(&[("/a.ts", "x\n")]); + let output = + grep(&call, &json!({ "pattern": "x", "glob": "*.{ts,tsx}" })).expect("the search succeeds"); + assert_eq!( + output.text(), + "No matches found. Searched 0 files.\n\n\ + [No ignore rules ran: .gitignore is not applied, and only entries named .git are \ + skipped.]\n\ + {, [ and ? in glob match themselves; there is no alternation or character class. Call \ + Grep once per alternative, or filter with type." + ); +} + +#[test] +fn a_bracket_or_a_question_mark_in_the_glob_also_adds_the_hint_and_a_plain_glob_does_not() { + let call = seeded(&[("/a.ts", "x\n")]); + for glob in ["*.[jt]s", "*.t?"] { + let output = + grep(&call, &json!({ "pattern": "x", "glob": glob })).expect("the search succeeds"); + assert!( + output + .text() + .contains("{, [ and ? in glob match themselves"), + "{glob}: {}", + output.text() + ); + } + let plain = + grep(&call, &json!({ "pattern": "zzz", "glob": "*.ts" })).expect("the search succeeds"); + assert!( + !plain.text().contains("match themselves"), + "{}", + plain.text() + ); +} + +#[test] +fn a_result_with_matches_carries_no_hint() { + let call = seeded(&[("/a.ts", "x\n")]); + let output = text_of(grep(&call, &json!({ "pattern": "x", "glob": "*.ts" }))); + assert_eq!(output, "/a.ts\n 1:x"); +} diff --git a/crates/plugin-std/src/grep/tests/paging.rs b/crates/plugin-std/src/grep/tests/paging.rs new file mode 100644 index 000000000..872cb50ed --- /dev/null +++ b/crates/plugin-std/src/grep/tests/paging.rs @@ -0,0 +1,157 @@ +//! Paging through a result with `head_limit` and `offset`, and the +//! limits that stop a search before the tree is done. + +use serde_json::json; + +use super::fake::FakeSource; +use super::{grep, grep_limited, grep_over, seeded, text_of}; +use crate::grep::found::Limits; + +/// Twenty-five lines, each a match. +fn twenty_five() -> String { + let lines: Vec = (1..=25).map(|number| format!("hit {number}\n")).collect(); + lines.concat() +} + +/// The page of `/f.txt` that `head_limit` and `offset` select. +fn page(call: &runvil_plugin::testing::TestCall, offset: usize) -> String { + text_of(grep( + call, + &json!({ "pattern": "hit", "head_limit": 10, "offset": offset }), + )) +} + +#[test] +fn three_pages_of_ten_say_where_to_continue_and_join_into_the_whole_result() { + let call = seeded(&[("/f.txt", &twenty_five())]); + let first = page(&call, 0); + let second = page(&call, 10); + let third = page(&call, 20); + assert!( + first.ends_with( + "[Showing 1-10 of at least 11 matching lines. Use offset=10 to see more, or narrow \ + the search.]" + ), + "{first}" + ); + assert!( + second.ends_with( + "[Showing 11-20 of at least 21 matching lines. Use offset=20 to see more, or narrow \ + the search.]" + ), + "{second}" + ); + assert!( + third.ends_with("[Showing 21-25 of 25 matching lines.]"), + "{third}" + ); + let rows = |text: &str| -> Vec { + text.lines() + .filter(|line| line.starts_with(" ")) + .map(str::to_owned) + .collect() + }; + let unpaged = text_of(grep(&call, &json!({ "pattern": "hit", "head_limit": 0 }))); + assert!(!unpaged.contains("[Showing"), "no notice without a limit"); + let joined: Vec = [&first, &second, &third] + .into_iter() + .flat_map(|text| rows(text)) + .collect(); + assert_eq!(joined, rows(&unpaged)); +} + +#[test] +fn an_offset_past_the_end_says_there_are_no_more_results() { + let call = seeded(&[("/f.txt", &twenty_five())]); + let past = grep(&call, &json!({ "pattern": "hit", "offset": 30 })).expect("it succeeds"); + assert_eq!(past.text(), "No more results at offset=30."); +} + +#[test] +fn a_write_between_two_page_calls_shows_in_the_second_because_nothing_is_cached() { + let call = seeded(&[("/f.txt", &twenty_five())]); + let first = page(&call, 0); + assert!(!first.contains("changed"), "{first}"); + let changed = twenty_five().replace("hit 12", "hit changed"); + call.context() + .access() + .write("/f.txt", changed.as_bytes()) + .expect("the file is rewritten"); + let second = page(&call, 10); + assert!(second.contains(" 12:hit changed"), "{second}"); +} + +#[test] +fn a_lowered_file_limit_stops_the_search_with_a_notice_that_offers_no_offset() { + let call = seeded(&[ + ("/a", "m\n"), + ("/b", "m\n"), + ("/c", "m\n"), + ("/d", "m\n"), + ("/e", "m\n"), + ]); + let limits = Limits { + max_files: 3, + ..Limits::default() + }; + let output = text_of(grep_limited(&call, &json!({ "pattern": "m" }), &limits)); + assert_eq!( + output, + "/a\n 1:m\n\n/b\n 1:m\n\n/c\n 1:m\n\n\ + [Search stopped after considering 3 files. Narrow path, glob or type to search the \ + rest.]" + ); +} + +#[test] +fn a_file_limit_that_is_not_passed_prints_no_notice() { + let call = seeded(&[("/a", "m\n"), ("/b", "m\n"), ("/c", "m\n")]); + let limits = Limits { + max_files: 3, + ..Limits::default() + }; + let output = text_of(grep_limited(&call, &json!({ "pattern": "m" }), &limits)); + assert!(!output.contains("[Search stopped"), "{output}"); +} + +#[test] +fn a_lowered_byte_limit_stops_the_search_before_the_next_read() { + let source = FakeSource::default() + .file("/a", "match one\n") + .file("/b", "match two\n") + .file("/c", "match three\n"); + let limits = Limits { + max_bytes: 15, + ..Limits::default() + }; + let output = text_of(grep_over(&source, &json!({ "pattern": "match" }), &limits)); + assert_eq!( + output, + "/a\n 1:match one\n\n/b\n 1:match two\n\n\ + [Search stopped after reading 15 bytes. Narrow path, glob or type to search the rest.]" + ); +} + +#[test] +fn a_limit_stop_with_no_match_still_says_so() { + let source = FakeSource::default().file("/a", "x\n").file("/b", "x\n"); + let limits = Limits { + max_files: 1, + ..Limits::default() + }; + let output = grep_over(&source, &json!({ "pattern": "zzz" }), &limits) + .expect("the search succeeds") + .text() + .to_owned(); + assert!( + output.starts_with("No matches found. Searched 1 file."), + "{output}" + ); + assert!( + output.ends_with( + "[Search stopped after considering 1 files. Narrow path, glob or type to search the \ + rest.]" + ), + "{output}" + ); +} diff --git a/crates/plugin-std/src/grep/tests/patterns.rs b/crates/plugin-std/src/grep/tests/patterns.rs new file mode 100644 index 000000000..fbdedbdc9 --- /dev/null +++ b/crates/plugin-std/src/grep/tests/patterns.rs @@ -0,0 +1,149 @@ +//! How Grep compiles and applies the pattern: the dialect, the flags, and +//! what a bad pattern says. + +use runvil_plugin::ToolErrorKind; +use serde_json::json; + +use super::{failure_of, grep, new_call, seeded, text_of}; + +/// The sentence every refused pattern ends with. +const ADVICE: &str = "Grep matches with Rust regex syntax, which has no look-around and no \ + back-references. To match the text literally, set literal to true."; + +#[test] +fn a_pattern_the_dialect_cannot_compile_is_an_error_naming_the_pattern_the_reason_and_the_dialect() +{ + let cases = [ + ( + "(?=x)y", + "look-around, including look-ahead and look-behind, is not supported", + ), + ( + "(?<=a)b", + "look-around, including look-ahead and look-behind, is not supported", + ), + ("(a)\\1", "backreferences are not supported"), + ("foo(", "unclosed group"), + ]; + for (pattern, reason) in cases { + let error = grep(&new_call(), &json!({ "pattern": pattern })) + .expect_err("the pattern does not compile"); + assert_eq!( + error.to_string(), + format!("std_Grep: Invalid regular expression \"{pattern}\": {reason}. {ADVICE}") + ); + assert_eq!(error.kind(), ToolErrorKind::InvalidArguments, "{pattern}"); + } +} + +#[test] +fn a_bad_pattern_fails_before_any_path_is_looked_at() { + let message = failure_of(grep( + &new_call(), + &json!({ "pattern": "foo(", "path": "/missing" }), + )); + assert!(message.contains("Invalid regular expression"), "{message}"); +} + +#[test] +fn a_pattern_that_holds_the_word_error_still_gets_its_own_reason() { + let message = failure_of(grep(&new_call(), &json!({ "pattern": "error: (" }))); + assert_eq!( + message, + format!("std_Grep: Invalid regular expression \"error: (\": unclosed group. {ADVICE}") + ); +} + +#[test] +fn literal_matches_the_pattern_as_plain_text() { + let call = seeded(&[("/one.txt", "axb\n"), ("/two.txt", "a.b\n")]); + let regex = text_of(grep( + &call, + &json!({ "pattern": "a.b", "output_mode": "count" }), + )); + assert_eq!( + regex, "/one.txt:1\n/two.txt:1", + "as a regex the dot matches x" + ); + let literal = text_of(grep( + &call, + &json!({ "pattern": "a.b", "literal": true, "output_mode": "count" }), + )); + assert_eq!(literal, "/two.txt:1"); +} + +#[test] +fn literal_makes_a_pattern_that_would_not_compile_match_as_text() { + let call = seeded(&[("/a.txt", "call foo( now\n")]); + let output = text_of(grep(&call, &json!({ "pattern": "foo(", "literal": true }))); + assert_eq!(output, "/a.txt\n 1:call foo( now"); +} + +#[test] +fn matching_is_case_sensitive_unless_i_is_set() { + let call = seeded(&[ + ("/a.txt", "Alpha\n"), + ("/b.txt", "alpha\n"), + ("/c.txt", "ALPHA\n"), + ]); + let exact = text_of(grep( + &call, + &json!({ "pattern": "alpha", "output_mode": "count" }), + )); + assert_eq!(exact, "/b.txt:1"); + let folded = text_of(grep( + &call, + &json!({ "pattern": "alpha", "-i": true, "output_mode": "count" }), + )); + assert_eq!(folded, "/a.txt:1\n/b.txt:1\n/c.txt:1"); +} + +#[test] +fn a_pattern_that_could_backtrack_forever_in_other_engines_completes_on_a_long_line() { + let long = "a".repeat(1_600_000); + let call = seeded(&[("/long.txt", &long), ("/almost.txt", &format!("{long}b"))]); + let output = text_of(grep( + &call, + &json!({ "pattern": "^(a|aa)+$", "output_mode": "count" }), + )); + assert_eq!(output, "/long.txt:1"); +} + +#[test] +fn multiline_lets_dot_cross_a_line_break_and_a_pattern_span_lines() { + let call = seeded(&[("/a.txt", "a\nb\n")]); + let spanning = text_of(grep(&call, &json!({ "pattern": "a.b", "multiline": true }))); + assert_eq!(spanning, "/a.txt\n 1:a\n 2:b"); + let plain = grep(&call, &json!({ "pattern": "a.b" })).expect("the search succeeds"); + assert_eq!( + plain.text(), + "No matches found. Searched 1 file.\n\n\ + [No ignore rules ran: .gitignore is not applied, and only entries named .git are \ + skipped.]" + ); +} + +#[test] +fn multiline_counts_every_line_a_match_touches_once() { + let call = seeded(&[("/a.txt", "x\na\nb\nc\nx\n")]); + let output = text_of(grep( + &call, + &json!({ "pattern": "a\nb\nc", "multiline": true, "output_mode": "count" }), + )); + assert_eq!(output, "/a.txt:3"); +} + +#[test] +fn carets_and_dollars_match_at_each_line() { + let call = seeded(&[("/a.txt", "one\ntwo\nthree\n")]); + let output = text_of(grep( + &call, + &json!({ "pattern": "^t", "output_mode": "count" }), + )); + assert_eq!(output, "/a.txt:2"); + let end = text_of(grep( + &call, + &json!({ "pattern": "e$", "output_mode": "count" }), + )); + assert_eq!(end, "/a.txt:2"); +} diff --git a/crates/plugin-std/src/grep/tests/render.rs b/crates/plugin-std/src/grep/tests/render.rs new file mode 100644 index 000000000..17ece3595 --- /dev/null +++ b/crates/plugin-std/src/grep/tests/render.rs @@ -0,0 +1,221 @@ +//! The text of a result: the worked example, context, gaps, the count +//! mode, and the byte cap. + +use serde_json::json; + +use super::{grep, grep_limited, seeded, text_of}; +use crate::grep::found::Limits; + +/// A file of `lines` lines where line `n` of `at` holds `hit n` and every +/// other line holds `.`. +fn file_with_hits(lines: usize, at: &[usize]) -> String { + (1..=lines) + .map(|number| { + if at.contains(&number) { + format!("hit {number}\n") + } else { + ".\n".to_owned() + } + }) + .collect() +} + +#[test] +fn the_worked_example_prints_paths_numbered_lines_a_gap_and_no_trailing_newline() { + let call = seeded(&[ + ( + "/src/a.ts", + "// one\n// two\nconst alpha = 1\nlet beta = 2\n// five\n// six\n// seven\n// eight\n\ + const gamma = 3\nexport default gamma\n", + ), + ("/src/b.ts", "const first = 0\n"), + ]); + let output = text_of(grep(&call, &json!({ "pattern": "const", "-A": 1 }))); + assert_eq!( + output, + "/src/a.ts\n 3:const alpha = 1\n 4-let beta = 2\n --\n 9:const gamma = 3\n \ + 10-export default gamma\n\n/src/b.ts\n 1:const first = 0" + ); + let tail = &output[output.len() - 30..]; + assert!(tail.ends_with("const first = 0"), "{tail:?}"); + assert!(!output.ends_with('\n')); +} + +#[test] +fn context_c_with_a_override_prints_each_groups_lines_with_a_gap_between() { + let call = seeded(&[("/a.txt", &file_with_hits(30, &[5, 20]))]); + let output = text_of(grep(&call, &json!({ "pattern": "hit", "-C": 3, "-A": 1 }))); + let numbers: Vec = output + .lines() + .skip(1) + .map(|line| line.trim_start().to_owned()) + .collect(); + assert_eq!( + numbers, + [ + "2-.", + "3-.", + "4-.", + "5:hit 5", + "6-.", + "--", + "17-.", + "18-.", + "19-.", + "20:hit 20", + "21-." + ] + ); +} + +#[test] +fn overlapping_context_merges_and_no_line_prints_twice() { + let call = seeded(&[("/a.txt", &file_with_hits(12, &[5, 7]))]); + let output = text_of(grep(&call, &json!({ "pattern": "hit", "-C": 2 }))); + let numbers: Vec = output + .lines() + .skip(1) + .map(|line| line.trim_start().to_owned()) + .collect(); + assert_eq!( + numbers, + ["3-.", "4-.", "5:hit 5", "6-.", "7:hit 7", "8-.", "9-."], + "lines 3 to 9 once, with no gap line" + ); +} + +#[test] +fn touching_context_merges_without_a_gap() { + let call = seeded(&[("/a.txt", &file_with_hits(12, &[3, 8]))]); + let output = text_of(grep(&call, &json!({ "pattern": "hit", "-A": 2, "-B": 2 }))); + assert!(!output.contains("--"), "{output}"); +} + +#[test] +fn without_context_two_far_apart_matches_have_no_gap_line_between_them() { + let call = seeded(&[("/a.txt", &file_with_hits(30, &[5, 20]))]); + let output = text_of(grep(&call, &json!({ "pattern": "hit" }))); + assert_eq!(output, "/a.txt\n 5:hit 5\n 20:hit 20"); +} + +#[test] +fn a_blank_line_separates_files_and_none_ends_the_output() { + let call = seeded(&[("/a.txt", "x\n"), ("/b.txt", "x\n")]); + let output = text_of(grep(&call, &json!({ "pattern": "x" }))); + assert_eq!(output, "/a.txt\n 1:x\n\n/b.txt\n 1:x"); +} + +#[test] +fn count_mode_prints_the_path_and_its_matching_line_count() { + let call = seeded(&[("/a.txt", "x\nx\ny\nx\n"), ("/b.txt", "y\n")]); + let output = text_of(grep( + &call, + &json!({ "pattern": "x", "output_mode": "count" }), + )); + assert_eq!(output, "/a.txt:3"); +} + +#[test] +fn count_mode_counts_after_the_last_colon_when_the_name_has_one() { + let call = seeded(&[("/a:b.txt", "x\nx\n")]); + let output = text_of(grep( + &call, + &json!({ "pattern": "x", "output_mode": "count" }), + )); + assert_eq!(output, "/a:b.txt:2"); +} + +#[test] +fn the_byte_cap_cuts_the_body_on_a_line_and_says_where_to_continue() { + let line = "x".repeat(100); + let text = format!("{line}\n").repeat(400); + let call = seeded(&[("/f.txt", &text)]); + let output = text_of(grep(&call, &json!({ "pattern": "x" }))); + let (body, notice) = output + .split_once("\n\n") + .expect("a body and a notice are separated by a blank line"); + assert!(body.len() <= 15_000, "{} bytes", body.len()); + assert!(body.ends_with(&line), "the body ends on a whole line"); + assert_eq!( + notice, + "[Showing 1-141 of at least 142 matching lines. Output stopped at 15000 bytes. Use \ + offset=141 to see more, or narrow the search.]" + ); +} + +#[test] +fn the_notice_after_a_cap_cut_names_the_offset_that_resumes_exactly_there() { + let line = "x".repeat(100); + let text = format!("{line}\n").repeat(400); + let call = seeded(&[("/f.txt", &text)]); + let next = text_of(grep(&call, &json!({ "pattern": "x", "offset": 141 }))); + assert!(next.starts_with("/f.txt\n 142:"), "{}", &next[..40]); +} + +#[test] +fn the_cap_applies_to_the_other_modes_and_counts_each_line_with_its_newline() { + let call = seeded(&[("/a", "x\n"), ("/b", "x\n"), ("/c", "x\n")]); + let limits = Limits { + body_cap: 6, + ..Limits::default() + }; + let output = text_of(grep_limited( + &call, + &json!({ "pattern": "x", "output_mode": "files_with_matches" }), + &limits, + )); + assert_eq!( + output, + "/a\n/b\n\n[Showing 1-2 of at least 3 files. Output stopped at 6 bytes. Use offset=2 to \ + see more, or narrow the search.]" + ); +} + +#[test] +fn a_cut_inside_a_files_lines_never_leaves_context_or_a_path_dangling() { + let call = seeded(&[("/a.txt", &file_with_hits(40, &[10, 30]))]); + for cap in 1..120 { + let limits = Limits { + body_cap: cap, + ..Limits::default() + }; + let output = text_of(grep_limited( + &call, + &json!({ "pattern": "hit", "-C": 2 }), + &limits, + )); + let body = output.split("\n\n").next().expect("a body"); + assert!( + body.len() < cap.max(1) || body.starts_with('['), + "cap {cap}: {output}" + ); + let last = body.lines().last().unwrap_or_default(); + assert!( + !last.starts_with('/') && last != " --", + "cap {cap} left {last:?} dangling: {output}" + ); + if let Some(first_row) = body.lines().nth(1) { + let leading_context = first_row.contains('-'); + let has_match = body.lines().any(|row| row.contains(":hit")); + assert!( + !leading_context || has_match, + "cap {cap} shows context with no match: {output}" + ); + } + } +} + +#[test] +fn a_result_whose_first_entry_cannot_fit_says_so_instead_of_printing_nothing() { + let call = seeded(&[("/a.txt", "x\n")]); + let limits = Limits { + body_cap: 3, + ..Limits::default() + }; + let output = text_of(grep_limited(&call, &json!({ "pattern": "x" }), &limits)); + assert_eq!( + output, + "[Output stopped at 3 bytes before the first result fit. Narrow the search with path, \ + glob or type.]" + ); +} diff --git a/crates/plugin-std/src/grep/tests/search.rs b/crates/plugin-std/src/grep/tests/search.rs new file mode 100644 index 000000000..ab8a335b7 --- /dev/null +++ b/crates/plugin-std/src/grep/tests/search.rs @@ -0,0 +1,233 @@ +//! Which files Grep visits and in what order, how a head limit and an +//! offset count across files, and what is trusted. + +use runvil_plugin::OutputTrust; +use serde_json::json; + +use super::{grep, new_call, seeded, text_of}; + +/// The files that sort differently by name and by full path. +const TRICKY: [(&str, &str); 5] = [ + ("/B", "x\n"), + ("/a-b", "x\n"), + ("/a.txt", "x\n"), + ("/a/x", "x\n"), + ("/b", "x\n"), +]; + +/// A file of `lines` lines where each line in `at` holds `hit` and every +/// other line holds `.`. +fn file_with_hits(lines: usize, at: &[usize]) -> String { + (1..=lines) + .map(|number| if at.contains(&number) { "hit\n" } else { ".\n" }) + .collect() +} + +#[test] +fn files_come_in_full_path_order_in_every_mode() { + let call = seeded(&TRICKY); + let content = text_of(grep(&call, &json!({ "pattern": "x" }))); + assert_eq!( + content, + "/B\n 1:x\n\n/a-b\n 1:x\n\n/a.txt\n 1:x\n\n/a/x\n 1:x\n\n/b\n 1:x" + ); + let files = text_of(grep( + &call, + &json!({ "pattern": "x", "output_mode": "files_with_matches" }), + )); + assert_eq!(files, "/B\n/a-b\n/a.txt\n/a/x\n/b"); + let count = text_of(grep( + &call, + &json!({ "pattern": "x", "output_mode": "count" }), + )); + assert_eq!(count, "/B:1\n/a-b:1\n/a.txt:1\n/a/x:1\n/b:1"); +} + +#[test] +fn the_order_is_the_order_glob_gives() { + let call = seeded(&TRICKY); + let globbed = call + .context() + .access() + .glob("/**") + .expect("the glob matches"); + let files: Vec = globbed + .into_iter() + .filter(|path| TRICKY.iter().any(|(file, _)| file == path)) + .collect(); + let searched = text_of(grep( + &call, + &json!({ "pattern": "x", "output_mode": "files_with_matches" }), + )); + assert_eq!(searched, files.join("\n")); +} + +#[test] +fn a_search_of_one_directory_prints_paths_as_absolute_virtual_paths() { + let call = seeded(&[("/proj/src/a.txt", "x\n"), ("/other.txt", "x\n")]); + for path in ["/proj", "/proj/", "proj"] { + let output = text_of(grep( + &call, + &json!({ "pattern": "x", "path": path, "output_mode": "files_with_matches" }), + )); + assert_eq!(output, "/proj/src/a.txt", "{path}"); + } +} + +#[test] +fn a_search_of_one_file_names_that_file() { + let call = seeded(&[("/proj/a.txt", "one\ntwo\n"), ("/proj/b.txt", "two\n")]); + let output = text_of(grep( + &call, + &json!({ "pattern": "two", "path": "/proj/a.txt" }), + )); + assert_eq!(output, "/proj/a.txt\n 2:two"); +} + +#[test] +fn entries_named_git_are_skipped_at_any_depth_and_other_dotfiles_are_searched() { + let call = seeded(&[ + ("/.git/config", "x\n"), + ("/sub/.git/HEAD", "x\n"), + ("/.gitignore", "x\n"), + ("/.env", "x\n"), + ("/target/out.txt", "x\n"), + ]); + let output = text_of(grep( + &call, + &json!({ "pattern": "x", "output_mode": "files_with_matches" }), + )); + assert_eq!(output, "/.env\n/.gitignore\n/target/out.txt"); +} + +#[test] +fn head_limit_counts_across_files_and_stops_inside_the_file_it_reaches() { + let call = seeded(&[ + ("/a.ts", &file_with_hits(45, &[12, 40])), + ("/b.ts", &file_with_hits(9, &[1, 2, 3])), + ]); + let output = text_of(grep(&call, &json!({ "pattern": "hit", "head_limit": 2 }))); + assert_eq!( + output, + "/a.ts\n 12:hit\n 40:hit\n\n\ + [Showing 1-2 of at least 3 matching lines. Use offset=2 to see more, or narrow the \ + search.]" + ); +} + +#[test] +fn an_offset_skips_entries_across_files_before_the_head_limit_applies() { + let call = seeded(&[ + ("/f1", "m\nm\nm\n"), + ("/f2", "m\nm\nm\n"), + ("/f3", "m\nm\nm\n"), + ]); + let output = text_of(grep( + &call, + &json!({ "pattern": "m", "offset": 4, "head_limit": 2 }), + )); + assert_eq!( + output, + "/f2\n 2:m\n 3:m\n\n\ + [Showing 5-6 of at least 7 matching lines. Use offset=6 to see more, or narrow the \ + search.]" + ); +} + +#[test] +fn head_limit_zero_prints_every_match() { + let text = "m\n".repeat(40); + let call = seeded(&[("/a.txt", &text)]); + let output = text_of(grep(&call, &json!({ "pattern": "m", "head_limit": 0 }))); + assert_eq!( + output.lines().count(), + 41, + "the path and 40 matches, no notice" + ); + assert!(output.ends_with("\n 40:m"), "{output}"); +} + +#[test] +fn a_head_limit_that_exactly_fits_prints_no_notice() { + let call = seeded(&[("/a.txt", "m\nm\n")]); + let output = text_of(grep(&call, &json!({ "pattern": "m", "head_limit": 2 }))); + assert_eq!(output, "/a.txt\n 1:m\n 2:m"); +} + +#[test] +fn head_limit_with_context_counts_entries_and_keeps_their_context_lines() { + let call = seeded(&[("/a.txt", &file_with_hits(30, &[1, 20, 28]))]); + let output = text_of(grep( + &call, + &json!({ "pattern": "hit", "-A": 5, "head_limit": 2 }), + )); + assert_eq!( + output, + "/a.txt\n 1:hit\n 2-.\n 3-.\n 4-.\n 5-.\n 6-.\n --\n 20:hit\n 21-.\n 22-.\n \ + 23-.\n 24-.\n 25-.\n\n\ + [Showing 1-2 of at least 3 matching lines. Use offset=2 to see more, or narrow the \ + search.]" + ); +} + +#[test] +fn files_with_matches_and_count_page_by_file() { + let call = seeded(&[("/a", "m\nm\n"), ("/b", "m\n"), ("/c", "m\n")]); + let files = text_of(grep( + &call, + &json!({ "pattern": "m", "output_mode": "files_with_matches", "head_limit": 2 }), + )); + assert_eq!( + files, + "/a\n/b\n\n[Showing 1-2 of at least 3 files. Use offset=2 to see more, or narrow the \ + search.]" + ); + let count = text_of(grep( + &call, + &json!({ "pattern": "m", "output_mode": "count", "offset": 1 }), + )); + assert_eq!(count, "/b:1\n/c:1\n\n[Showing 2-3 of 3 files.]"); +} + +#[test] +fn a_result_with_matches_is_untrusted_output() { + let call = seeded(&[("/a.txt", "needle\n")]); + let output = grep(&call, &json!({ "pattern": "needle" })).expect("the search succeeds"); + assert_eq!(output.trust(), OutputTrust::Untrusted); +} + +#[test] +fn a_bare_zero_match_result_is_trusted_and_a_past_the_end_one_is_not() { + let call = seeded(&[("/a.txt", "needle\n")]); + let none = grep(&call, &json!({ "pattern": "absent" })).expect("the search succeeds"); + assert_eq!(none.trust(), OutputTrust::Trusted); + assert_eq!( + none.text(), + "No matches found. Searched 1 file.\n\n\ + [No ignore rules ran: .gitignore is not applied, and only entries named .git are \ + skipped.]" + ); + let past = + grep(&call, &json!({ "pattern": "needle", "offset": 5 })).expect("the search succeeds"); + assert_eq!(past.text(), "No more results at offset=5."); + assert_eq!(past.trust(), OutputTrust::Untrusted); +} + +#[test] +fn searching_nothing_says_no_files_were_searched() { + let output = grep(&new_call(), &json!({ "pattern": "x" })).expect("the search succeeds"); + assert!( + output + .text() + .starts_with("No matches found. Searched 0 files."), + "{}", + output.text() + ); +} + +#[test] +fn the_description_names_the_file_limit_the_code_uses() { + let text = &super::definition().description; + let limit = crate::grep::found::MAX_FILES.to_string(); + assert!(text.contains(&limit), "{limit} is named: {text}"); +} diff --git a/crates/plugin-std/src/grep/tests/skips.rs b/crates/plugin-std/src/grep/tests/skips.rs new file mode 100644 index 000000000..8f1662fcb --- /dev/null +++ b/crates/plugin-std/src/grep/tests/skips.rs @@ -0,0 +1,390 @@ +//! What Grep skips and counts instead of failing, and how the limits and +//! the pauses behave, through a [`Source`](crate::grep::source::Source) +//! that can produce what the in-memory backend cannot. + +use runvil_plugin::{OutputTrust, VfsError}; +use serde_json::json; + +use super::fake::{FakeSource, Node}; +use super::{block_on, count_pendings, definition, failure_of, grep_over, text_of}; +use crate::grep::args; +use crate::grep::found::Limits; + +/// The error for a path the policy refuses. +fn denied(path: &str) -> VfsError { + VfsError::PermissionDenied { + path: path.to_owned(), + reason: "the policy refuses it".to_owned(), + } +} + +/// A tree with one of everything Grep skips, and one clean match. +fn messy() -> FakeSource { + FakeSource::default() + .with("/a.bin", Node::File(b"NEEDLE\0\0".to_vec())) + .file("/bad\u{1}name", "NEEDLE\n") + .with("/big.log", Node::Sized(1000, b"NEEDLE\n".to_vec())) + .with( + "/busy.txt", + Node::ReadFails(VfsError::Conflict { + path: "/busy.txt".to_owned(), + detail: "another call".to_owned(), + }), + ) + .with("/denied.txt", Node::ReadFails(denied("/denied.txt"))) + .with("/deny", Node::ListFails(denied("/deny"))) + .with( + "/io.txt", + Node::ReadFails(VfsError::Io { + path: "/io.txt".to_owned(), + kind: std::io::ErrorKind::Other, + raw_os_error: None, + message: "disk error".to_owned(), + }), + ) + .with("/link", Node::Link) + .with("/pipe", Node::Pipe) + .file("/ok.txt", "NEEDLE here\n") +} + +/// The limits the skip tests lower the file size under. +fn small_files() -> Limits { + Limits { + max_file_size: 100, + ..Limits::default() + } +} + +/// The skip line every `messy` search ends with. +const MESSY_SKIPS: &str = "[Skipped 9 entries: 1 binary file, 2 files that could not be read, 1 \ + file over 100 bytes, 1 symbolic link, 1 name with a control character, 1 entry held by \ + another call in this round, 2 entries denied by policy. First: /a.bin, /big.log, \ + /busy.txt, /denied.txt, /deny]"; + +#[test] +fn every_kind_of_skip_is_counted_and_the_clean_match_is_still_found() { + let output = text_of(grep_over( + &messy(), + &json!({ "pattern": "NEEDLE" }), + &small_files(), + )); + assert_eq!(output, format!("/ok.txt\n 1:NEEDLE here\n\n{MESSY_SKIPS}")); +} + +#[test] +fn a_zero_match_result_keeps_the_skip_line_and_the_searched_count() { + let output = grep_over(&messy(), &json!({ "pattern": "absent" }), &small_files()) + .expect("the search succeeds"); + assert_eq!( + output.trust(), + OutputTrust::Untrusted, + "it names skipped paths" + ); + assert_eq!( + output.text(), + format!( + "No matches found. Searched 1 file.\n\n{MESSY_SKIPS}\n\ + [No ignore rules ran: .gitignore is not applied, and only entries named .git are \ + skipped.]" + ) + ); +} + +#[test] +fn a_binary_file_is_skipped_in_every_output_mode() { + let source = FakeSource::default().with("/a.bin", Node::File(b"NEEDLE\0".to_vec())); + for mode in ["content", "files_with_matches", "count"] { + let output = grep_over( + &source, + &json!({ "pattern": "NEEDLE", "output_mode": mode }), + &Limits::default(), + ) + .expect("the search succeeds"); + assert_eq!( + output.text(), + "No matches found. Searched 0 files.\n\n\ + [Skipped 1 entry: 1 binary file. First: /a.bin]\n\ + [No ignore rules ran: .gitignore is not applied, and only entries named .git are \ + skipped.]", + "{mode}" + ); + } +} + +#[test] +fn the_skip_line_for_one_skip_uses_the_singular() { + let source = FakeSource::default() + .with("/a.bin", Node::File(b"x\0".to_vec())) + .file("/ok.txt", "NEEDLE\n"); + let output = text_of(grep_over( + &source, + &json!({ "pattern": "NEEDLE" }), + &Limits::default(), + )); + assert_eq!( + output, + "/ok.txt\n 1:NEEDLE\n\n[Skipped 1 entry: 1 binary file. First: /a.bin]" + ); +} + +#[test] +fn a_control_character_name_is_counted_and_never_printed() { + let source = FakeSource::default() + .file("/a\u{7f}b", "NEEDLE\n") + .file("/ok.txt", "NEEDLE\n"); + let output = text_of(grep_over( + &source, + &json!({ "pattern": "NEEDLE" }), + &Limits::default(), + )); + assert_eq!( + output, + "/ok.txt\n 1:NEEDLE\n\n[Skipped 1 entry: 1 name with a control character.]" + ); +} + +#[test] +fn only_the_first_five_printable_skips_are_named() { + let mut source = FakeSource::default(); + for index in 0..8 { + source = source.with(&format!("/f{index}.bin"), Node::File(b"\0".to_vec())); + } + let output = grep_over(&source, &json!({ "pattern": "x" }), &Limits::default()) + .expect("the search succeeds"); + assert!( + output.text().contains( + "[Skipped 8 entries: 8 binary files. First: /f0.bin, /f1.bin, /f2.bin, /f3.bin, \ + /f4.bin]" + ), + "{}", + output.text() + ); +} + +#[test] +fn a_directory_that_cannot_be_listed_is_skipped_by_why() { + let source = FakeSource::default() + .with( + "/io", + Node::ListFails(VfsError::Io { + path: "/io".to_owned(), + kind: std::io::ErrorKind::Other, + raw_os_error: None, + message: "disk error".to_owned(), + }), + ) + .with( + "/locked", + Node::ListFails(VfsError::Conflict { + path: "/locked".to_owned(), + detail: "another call".to_owned(), + }), + ) + .with("/policy", Node::ListFails(denied("/policy"))) + .file("/ok.txt", "NEEDLE\n"); + let output = text_of(grep_over( + &source, + &json!({ "pattern": "NEEDLE" }), + &Limits::default(), + )); + assert_eq!( + output, + "/ok.txt\n 1:NEEDLE\n\n[Skipped 3 entries: 1 directory that could not be read, 1 entry \ + held by another call in this round, 1 entry denied by policy. First: /io, /locked, \ + /policy]" + ); +} + +#[test] +fn a_pipe_is_never_opened_and_is_skipped_as_unreadable() { + let source = FakeSource::default().with("/pipe", Node::Pipe); + let output = grep_over(&source, &json!({ "pattern": "x" }), &Limits::default()) + .expect("the search succeeds"); + assert!( + output + .text() + .contains("[Skipped 1 entry: 1 file that could not be read. First: /pipe]"), + "{}", + output.text() + ); +} + +#[test] +fn a_file_over_the_size_limit_is_skipped_before_any_read() { + let source = FakeSource::default().with("/big.log", Node::Sized(1000, b"NEEDLE\n".to_vec())); + let output = grep_over(&source, &json!({ "pattern": "NEEDLE" }), &small_files()) + .expect("the search succeeds"); + assert!( + output + .text() + .contains("[Skipped 1 entry: 1 file over 100 bytes. First: /big.log]"), + "{}", + output.text() + ); +} + +#[test] +fn a_missing_path_fails_with_the_does_not_exist_text() { + let source = FakeSource::default().file("/a.txt", "x\n"); + let message = failure_of(grep_over( + &source, + &json!({ "pattern": "x", "path": "/x" }), + &Limits::default(), + )); + assert_eq!(message, "Path does not exist: /x"); +} + +#[test] +fn a_root_that_cannot_be_listed_or_statted_fails_through_the_refusal_text() { + let source = FakeSource::default().with("/deny", Node::ListFails(denied("/deny"))); + let message = failure_of(grep_over( + &source, + &json!({ "pattern": "x", "path": "/deny" }), + &Limits::default(), + )); + assert_eq!(message, "the policy refuses it"); +} + +#[test] +fn a_root_that_is_a_pipe_is_neither_a_file_nor_a_directory() { + let source = FakeSource::default().with("/pipe", Node::Pipe); + let message = failure_of(grep_over( + &source, + &json!({ "pattern": "x", "path": "/pipe" }), + &Limits::default(), + )); + assert_eq!(message, "Path is neither a file nor a directory: /pipe"); +} + +#[test] +fn a_named_file_is_searched_alone_and_a_failed_read_of_it_fails_the_call() { + let source = FakeSource::default() + .file("/a.txt", "NEEDLE\n") + .with("/denied.txt", Node::ReadFails(denied("/denied.txt"))); + let output = text_of(grep_over( + &source, + &json!({ "pattern": "NEEDLE", "path": "/a.txt" }), + &Limits::default(), + )); + assert_eq!(output, "/a.txt\n 1:NEEDLE"); + let message = failure_of(grep_over( + &source, + &json!({ "pattern": "NEEDLE", "path": "/denied.txt" }), + &Limits::default(), + )); + assert_eq!(message, "the policy refuses it"); +} + +#[test] +fn a_named_file_over_the_size_limit_is_a_skip() { + let source = FakeSource::default().with("/big", Node::Sized(1000, b"NEEDLE\n".to_vec())); + let output = grep_over( + &source, + &json!({ "pattern": "NEEDLE", "path": "/big" }), + &small_files(), + ) + .expect("the search succeeds"); + assert!( + output + .text() + .starts_with("No matches found. Searched 0 files.\n\n[Skipped 1 entry: 1 file over 100 bytes. First: /big]"), + "{}", + output.text() + ); +} + +#[test] +fn a_named_link_is_searched_through_as_a_directory_and_then_as_a_file() { + let source = FakeSource::default() + .with("/dirlink", Node::Link) + .file("/dirlink/a.txt", "NEEDLE\n") + .with("/filelink", Node::LinkToFile(b"NEEDLE inside\n".to_vec())); + let through_dir = text_of(grep_over( + &source, + &json!({ "pattern": "NEEDLE", "path": "/dirlink" }), + &Limits::default(), + )); + assert_eq!(through_dir, "/dirlink/a.txt\n 1:NEEDLE"); + let through_file = text_of(grep_over( + &source, + &json!({ "pattern": "NEEDLE", "path": "/filelink" }), + &Limits::default(), + )); + assert_eq!(through_file, "/filelink\n 1:NEEDLE inside"); +} + +#[test] +fn a_named_link_to_an_over_size_file_is_a_skip_found_after_a_bounded_read() { + let source = FakeSource::default().with("/filelink", Node::LinkToFile(vec![b'a'; 500])); + let output = grep_over( + &source, + &json!({ "pattern": "a", "path": "/filelink" }), + &small_files(), + ) + .expect("the search succeeds"); + assert!( + output + .text() + .contains("[Skipped 1 entry: 1 file over 100 bytes. First: /filelink]"), + "{}", + output.text() + ); +} + +#[test] +fn the_search_pauses_once_per_slice_of_files() { + let mut source = FakeSource::default(); + for index in 0..10 { + source = source.file(&format!("/f{index}"), "x\n"); + } + let limits = Limits { + slice_files: 3, + ..Limits::default() + }; + let definition = definition(); + let args = json!({ "pattern": "x", "output_mode": "count" }); + let request = args::parse(definition, &args).expect("the arguments parse"); + let (output, pending) = count_pendings(crate::grep::search(&source, "Grep", &request, &limits)); + assert!(output.is_ok()); + assert_eq!(pending, 3, "after files 3, 6, and 9"); +} + +#[test] +fn the_search_pauses_once_per_slice_of_bytes() { + let mut source = FakeSource::default(); + for index in 0..10 { + source = source.file(&format!("/f{index}"), "0123456789"); + } + let limits = Limits { + slice_bytes: 25, + ..Limits::default() + }; + let definition = definition(); + let args = json!({ "pattern": "x" }); + let request = args::parse(definition, &args).expect("the arguments parse"); + let (output, pending) = count_pendings(crate::grep::search(&source, "Grep", &request, &limits)); + assert!(output.is_ok()); + assert_eq!(pending, 3, "after files 3, 6, and 9, at 30 bytes each time"); +} + +#[test] +fn a_search_dropped_after_its_first_pause_leaves_the_source_usable() { + let mut source = FakeSource::default(); + for index in 0..10 { + source = source.file(&format!("/f{index}"), "x\n"); + } + let limits = Limits { + slice_files: 2, + ..Limits::default() + }; + let definition = definition(); + let args = json!({ "pattern": "x" }); + let request = args::parse(definition, &args).expect("the arguments parse"); + { + let mut future = Box::pin(crate::grep::search(&source, "Grep", &request, &limits)); + let mut cx = std::task::Context::from_waker(std::task::Waker::noop()); + assert!(future.as_mut().poll(&mut cx).is_pending()); + } + let again = block_on(crate::grep::search(&source, "Grep", &request, &limits)); + assert!(again.is_ok(), "a fresh search still runs"); +} diff --git a/crates/plugin-std/src/grep/tests/text.rs b/crates/plugin-std/src/grep/tests/text.rs new file mode 100644 index 000000000..7f386bae8 --- /dev/null +++ b/crates/plugin-std/src/grep/tests/text.rs @@ -0,0 +1,197 @@ +//! How Grep reads a file's bytes: through the same `text` module Read +//! uses, so a Grep line number is the number Read prints for that line. + +use runvil_plugin::OutputTrust; +use serde_json::json; + +use super::fake::{FakeSource, Node}; +use super::{grep_over, text_of}; +use crate::grep::found::Limits; + +/// The tree with `bytes` as the file `/a.txt`. +fn one(bytes: &[u8]) -> FakeSource { + FakeSource::default().with("/a.txt", Node::File(bytes.to_vec())) +} + +/// The text of searching `source` for `pattern`. +fn search(source: &FakeSource, pattern: &str) -> String { + text_of(grep_over( + source, + &json!({ "pattern": pattern }), + &Limits::default(), + )) +} + +/// `text` as UTF-16 little-endian code units, with a byte order mark when +/// `mark` is set. +fn utf16le(text: &str, mark: bool) -> Vec { + let mut bytes = if mark { vec![0xFF, 0xFE] } else { Vec::new() }; + for unit in text.encode_utf16() { + bytes.extend_from_slice(&unit.to_le_bytes()); + } + bytes +} + +#[test] +fn invalid_utf8_matches_with_the_replacement_character_and_a_note() { + let output = search(&one(b"caf\xE9 NEEDLE\n"), "NEEDLE"); + assert_eq!( + output, + "/a.txt\n 1:caf\u{FFFD} NEEDLE\n\n\ + [Note: invalid UTF-8 in 1 file was shown as U+FFFD: /a.txt]" + ); +} + +#[test] +fn a_byte_order_mark_is_dropped_so_a_caret_matches_the_first_line() { + let mut bytes = vec![0xEF, 0xBB, 0xBF]; + bytes.extend_from_slice(b"NEEDLE first\nsecond\n"); + let output = search(&one(&bytes), "^NEEDLE"); + assert_eq!(output, "/a.txt\n 1:NEEDLE first"); +} + +#[test] +fn a_carriage_return_before_a_line_feed_is_not_part_of_the_line() { + let output = search(&one(b"a\r\nNEEDLE\r\n"), "NEEDLE"); + assert_eq!(output, "/a.txt\n 2:NEEDLE"); + let anchored = search(&one(b"a\r\nNEEDLE\r\n"), "NEEDLE$"); + assert_eq!(anchored, "/a.txt\n 2:NEEDLE"); +} + +#[test] +fn a_lone_carriage_return_does_not_start_a_line() { + let output = search(&one(b"first\rNEEDLE\rthird"), "NEEDLE"); + assert_eq!(output, "/a.txt\n 1:first\rNEEDLE\rthird"); +} + +#[test] +fn utf16_with_a_mark_is_decoded_and_numbered_by_its_lines() { + let output = search(&one(&utf16le("a\nNEEDLE\n", true)), "NEEDLE"); + assert_eq!(output, "/a.txt\n 2:NEEDLE"); +} + +#[test] +fn utf16_without_a_mark_is_one_binary_skip() { + let output = grep_over( + &one(&utf16le("NEEDLE\n", false)), + &json!({ "pattern": "NEEDLE" }), + &Limits::default(), + ) + .expect("the search succeeds"); + assert!( + output + .text() + .contains("[Skipped 1 entry: 1 binary file. First: /a.txt]"), + "{}", + output.text() + ); +} + +#[test] +fn five_control_bytes_in_a_hundred_make_a_file_binary_and_four_do_not() { + let body = |controls: usize| { + let mut bytes = vec![0x01; controls]; + bytes.extend_from_slice(b"NEEDLE"); + bytes.resize(100, b'a'); + bytes + }; + let binary = grep_over( + &one(&body(5)), + &json!({ "pattern": "NEEDLE" }), + &Limits::default(), + ) + .expect("the search succeeds"); + assert!(binary.text().contains("1 binary file"), "{}", binary.text()); + let text = grep_over( + &one(&body(4)), + &json!({ "pattern": "NEEDLE", "output_mode": "files_with_matches" }), + &Limits::default(), + ) + .expect("the search succeeds"); + assert_eq!(text.text(), "/a.txt"); +} + +#[test] +fn a_precomposed_character_and_a_combining_pair_each_match_only_their_own_file() { + let source = FakeSource::default() + .file("/one.txt", "caf\u{e9}\n") + .file("/two.txt", "cafe\u{301}\n"); + let precomposed = text_of(grep_over( + &source, + &json!({ "pattern": "caf\u{e9}", "output_mode": "files_with_matches" }), + &Limits::default(), + )); + assert_eq!(precomposed, "/one.txt"); + let combining = text_of(grep_over( + &source, + &json!({ "pattern": "cafe\u{301}", "output_mode": "files_with_matches" }), + &Limits::default(), + )); + assert_eq!(combining, "/two.txt"); +} + +#[test] +fn two_replaced_files_are_named_in_one_note() { + let source = FakeSource::default() + .with("/a.txt", Node::File(b"x \xFF\n".to_vec())) + .with("/b.txt", Node::File(b"x \xFE\n".to_vec())); + let output = text_of(grep_over( + &source, + &json!({ "pattern": "x", "output_mode": "files_with_matches" }), + &Limits::default(), + )); + assert_eq!( + output, + "/a.txt\n/b.txt\n\n\ + [Note: invalid UTF-8 in 2 files was shown as U+FFFD: /a.txt, /b.txt]" + ); +} + +#[test] +fn a_note_names_the_first_five_files_and_counts_the_rest() { + let mut source = FakeSource::default(); + for index in 0..7 { + source = source.with(&format!("/f{index}.txt"), Node::File(b"x \xFF\n".to_vec())); + } + let output = text_of(grep_over( + &source, + &json!({ "pattern": "x", "output_mode": "count" }), + &Limits::default(), + )); + assert!( + output.ends_with( + "[Note: invalid UTF-8 in 7 files was shown as U+FFFD: /f0.txt, /f1.txt, /f2.txt, \ + /f3.txt, /f4.txt, and 2 more]" + ), + "{output}" + ); +} + +#[test] +fn a_replacement_note_on_a_zero_match_result_makes_it_untrusted() { + let output = grep_over( + &one(b"caf\xE9\n"), + &json!({ "pattern": "absent" }), + &Limits::default(), + ) + .expect("the search succeeds"); + assert_eq!(output.trust(), OutputTrust::Untrusted); + assert!( + output.text().contains("[Note: invalid UTF-8 in 1 file"), + "{}", + output.text() + ); +} + +#[test] +fn an_empty_file_has_no_lines_to_match() { + let output = grep_over(&one(b""), &json!({ "pattern": "^" }), &Limits::default()) + .expect("the search succeeds"); + assert!( + output + .text() + .starts_with("No matches found. Searched 1 file."), + "{}", + output.text() + ); +} diff --git a/crates/plugin-std/src/grep/tests/window.rs b/crates/plugin-std/src/grep/tests/window.rs new file mode 100644 index 000000000..7134d339d --- /dev/null +++ b/crates/plugin-std/src/grep/tests/window.rs @@ -0,0 +1,199 @@ +//! The two cuts inside a line: the window around a match and the head of +//! a context line. No cut may split a code point, and the omitted counts +//! plus what is shown must equal the line. + +use serde_json::json; + +use super::{grep_limited, seeded, text_of}; +use crate::grep::found::Limits; +use crate::grep::window::{Window, context_text, match_text}; + +/// Splits shown text into the bytes omitted before it, the window, and +/// the bytes omitted after it. +fn parts(shown: &str) -> (usize, &str, usize) { + let (before, rest) = match shown.strip_prefix("[... ") { + Some(after_open) => { + let (count, rest) = after_open + .split_once(" bytes omitted]") + .expect("an opening marker closes"); + (count.parse().expect("the count is a number"), rest) + } + None => (0, shown), + }; + match rest.rfind("[... ") { + Some(open) if rest.ends_with(" bytes omitted]") => { + let count = rest[open + "[... ".len()..] + .strip_suffix(" bytes omitted]") + .expect("the marker closes the text") + .parse() + .expect("the count is a number"); + (before, &rest[..open], count) + } + _ => (before, rest, 0), + } +} + +/// Asserts that `shown` is a well-formed cut of `line`: the window is a +/// whole-character slice of it at the offset the first count gives, and +/// the counts plus the window add up to the line. +fn assert_cut_of(line: &str, shown: &str) { + let (before, window, after) = parts(shown); + assert_eq!( + before + window.len() + after, + line.len(), + "the counts and the window cover the line: {shown:?}" + ); + assert_eq!(line.get(before..before + window.len()), Some(window)); +} + +#[test] +fn a_line_within_the_width_is_shown_whole() { + let window = Window::default(); + let line = "y".repeat(1000); + assert_eq!(match_text(&line, 500, &window), line); + assert_eq!(context_text(&line, &window), line); +} + +#[test] +fn a_match_past_the_first_thousand_bytes_is_still_shown_with_both_markers() { + let line = format!("{}NEEDLE{}", "a".repeat(1001), "a".repeat(993)); + assert_eq!(line.len(), 2000); + let shown = match_text(&line, 1001, &Window::default()); + assert!(shown.contains("NEEDLE"), "{shown}"); + assert!(shown.starts_with("[... 801 bytes omitted]"), "{shown}"); + assert!(shown.ends_with("[... 199 bytes omitted]"), "{shown}"); + let (before, window, after) = parts(&shown); + assert_eq!((before, window.len(), after), (801, 1000, 199)); + assert_eq!(before + window.len() + after, line.len()); +} + +#[test] +fn a_match_near_the_start_has_only_a_trailing_marker_and_one_near_the_end_only_a_leading() { + let line = format!("NEEDLE{}", "a".repeat(2000)); + let head = match_text(&line, 0, &Window::default()); + assert!(!head.starts_with("[... "), "{head}"); + assert_cut_of(&line, &head); + let end = "a".repeat(2000) + "NEEDLE"; + let tail = match_text(&end, 2000, &Window::default()); + assert!(tail.starts_with("[... "), "{tail}"); + assert!(!tail.ends_with(" bytes omitted]"), "{tail}"); + assert!(tail.contains("NEEDLE"), "{tail}"); + assert_cut_of(&end, &tail); +} + +#[test] +fn a_line_of_cjk_characters_is_cut_on_character_boundaries() { + let line = "日本語".repeat(200); + assert_eq!(line.chars().count(), 600); + for first in [0, 700, 899, 1000, 1801] { + let shown = match_text(&line, first, &Window::default()); + assert_cut_of(&line, &shown); + } +} + +#[test] +fn a_line_of_four_byte_emoji_with_the_match_in_the_middle_is_cut_on_character_boundaries() { + let line = "😀".repeat(1000); + assert_eq!(line.len(), 4000); + for first in 2000..2004 { + let shown = match_text(&line, first, &Window::default()); + assert_cut_of(&line, &shown); + let (_, window, _) = parts(&shown); + assert!(window.chars().all(|c| c == '😀'), "whole characters only"); + } +} + +#[test] +fn a_context_line_over_the_width_keeps_its_head_and_counts_the_rest() { + let line = "日".repeat(600); + let shown = context_text(&line, &Window::default()); + let (before, window, after) = parts(&shown); + assert_eq!(before, 0); + assert!(window.len() <= 1000, "{} bytes", window.len()); + assert_eq!(window.len(), 999, "333 whole characters"); + assert_eq!(after, 1800 - 999); + assert_cut_of(&line, &shown); +} + +#[test] +fn sweeping_the_window_start_over_every_byte_of_a_three_byte_character_never_splits_it() { + let line = format!("{}日{}", "x".repeat(1500), "y".repeat(1500)); + for first in 1690..1710 { + let shown = match_text(&line, first, &Window::default()); + assert_cut_of(&line, &shown); + } +} + +#[test] +fn sweeping_the_width_and_lead_over_multibyte_text_keeps_every_cut_whole() { + let lines = [ + "日本語のテキスト".repeat(5), + "😀🎉".repeat(12), + format!("é{}ß{}", "a".repeat(9), "日".repeat(9)), + ]; + for line in &lines { + for width in 1..30 { + for lead in [0, 1, 2, 3, 5, 8] { + let window = Window { width, lead }; + for first in 0..line.len() { + assert_cut_of(line, &match_text(line, first, &window)); + } + assert_cut_of(line, &context_text(line, &window)); + } + } + } +} + +#[test] +fn sweeping_the_body_cap_and_window_over_emoji_and_cjk_text_keeps_the_body_whole_lines() { + let text = format!( + "{}\n{}\n{}\n", + "日本語".repeat(12), + "😀 needle 🎉".repeat(4), + "needle é" + ); + let call = seeded(&[("/a.txt", &text)]); + for width in [1, 2, 3, 4, 7, 10, 25] { + for lead in [0, 1, 4] { + for cap in (10..260).step_by(13) { + let limits = Limits { + body_cap: cap, + window: Window { width, lead }, + ..Limits::default() + }; + let output = text_of(grep_limited( + &call, + &json!({ "pattern": "needle", "-C": 1 }), + &limits, + )); + let body = output.split("\n\n").next().expect("a body"); + assert!( + body.len() <= cap || body.starts_with('['), + "cap {cap} width {width}: {output}" + ); + for row in body.lines() { + let numbered = row + .strip_prefix(" ") + .is_some_and(|rest| rest.starts_with(|c: char| c.is_ascii_digit())); + assert!( + row.starts_with('/') || row == " --" || numbered || row.starts_with('['), + "cap {cap} width {width}: {row:?} is not a whole row in {output}" + ); + } + } + } + } +} + +#[test] +fn a_long_matching_line_in_a_search_is_shown_as_a_window_with_its_markers() { + let line = format!("{}NEEDLE{}", "a".repeat(1500), "b".repeat(1500)); + let call = seeded(&[("/long.txt", &format!("{line}\n"))]); + let output = text_of(super::grep(&call, &json!({ "pattern": "NEEDLE" }))); + let row = output.lines().nth(1).expect("a match row"); + let shown = row.strip_prefix(" 1:").expect("the row is numbered"); + assert!(shown.contains("NEEDLE"), "{shown}"); + assert_cut_of(&line, shown); + let (before, _, _) = parts(shown); + assert_eq!(before, 1300, "the window starts 200 bytes before the match"); +} diff --git a/crates/plugin-std/src/grep/types.rs b/crates/plugin-std/src/grep/types.rs new file mode 100644 index 000000000..dca644961 --- /dev/null +++ b/crates/plugin-std/src/grep/types.rs @@ -0,0 +1,49 @@ +//! Grep's `type` filter: a short name for a set of file extensions. + +/// Each type name beside the extensions it keeps, without the dot. +const TYPES: [(&str, &[&str]); 16] = [ + ("js", &["js", "mjs", "cjs", "jsx"]), + ("ts", &["ts", "tsx", "mts", "cts"]), + ("py", &["py", "pyi"]), + ("rust", &["rs"]), + ("go", &["go"]), + ("java", &["java"]), + ("c", &["c", "h"]), + ("cpp", &["cc", "cpp", "cxx", "hh", "hpp", "hxx", "h"]), + ("cs", &["cs"]), + ("rb", &["rb"]), + ("php", &["php"]), + ("md", &["md", "markdown"]), + ("json", &["json"]), + ("yaml", &["yaml", "yml"]), + ("toml", &["toml"]), + ("html", &["html", "htm"]), +]; + +/// The extensions of the type `name`, or `None` when no type has that +/// name. Names are case-sensitive. +pub(super) fn extensions(name: &str) -> Option<&'static [&'static str]> { + TYPES + .iter() + .find(|(known, _)| *known == name) + .map(|(_, extensions)| *extensions) +} + +/// The extension of `file_name`: the text after its last `.`, unless that +/// dot is the first character, so `a.js` has `js` and `.js` has none. +fn extension(file_name: &str) -> Option<&str> { + let dot = file_name.rfind('.')?; + if dot == 0 { + return None; + } + file_name.get(dot + 1..) +} + +/// Whether the file called `file_name` has one of `extensions`, compared +/// case-sensitively. +pub(super) fn matches(extensions: &[&str], file_name: &str) -> bool { + extension(file_name).is_some_and(|found| extensions.contains(&found)) +} + +#[cfg(test)] +mod tests; diff --git a/crates/plugin-std/src/grep/types/tests.rs b/crates/plugin-std/src/grep/types/tests.rs new file mode 100644 index 000000000..b56734a35 --- /dev/null +++ b/crates/plugin-std/src/grep/types/tests.rs @@ -0,0 +1,34 @@ +//! Unit tests for the `type` table and the extension rule. + +use super::{extension, extensions, matches}; + +#[test] +fn the_extension_is_the_text_after_the_last_dot_unless_that_dot_is_first() { + assert_eq!(extension("a.js"), Some("js")); + assert_eq!(extension("a.b.js"), Some("js")); + assert_eq!(extension(".js"), None); + assert_eq!(extension("Makefile"), None); + assert_eq!(extension("x."), Some("")); +} + +#[test] +fn a_type_keeps_its_extensions_and_matching_is_case_sensitive() { + let rust = extensions("rust").expect("rust is a type"); + assert!(matches(rust, "lib.rs")); + assert!(!matches(rust, "lib.RS")); + assert!(!matches(rust, "rs")); + let cpp = extensions("cpp").expect("cpp is a type"); + assert!(matches(cpp, "a.h") && matches(cpp, "a.hpp")); + assert!(extensions("Rust").is_none()); + assert!(extensions("foo").is_none()); +} + +#[test] +fn every_documented_type_is_known() { + for name in [ + "js", "ts", "py", "rust", "go", "java", "c", "cpp", "cs", "rb", "php", "md", "json", + "yaml", "toml", "html", + ] { + assert!(extensions(name).is_some(), "{name}"); + } +} diff --git a/crates/plugin-std/src/grep/window.rs b/crates/plugin-std/src/grep/window.rs new file mode 100644 index 000000000..8ed11cb68 --- /dev/null +++ b/crates/plugin-std/src/grep/window.rs @@ -0,0 +1,76 @@ +//! The two cuts Grep makes inside a line: a window around a match, and +//! the head of a context line. +//! +//! Both cuts go through `clip_to_boundary`, so neither splits a code +//! point, and both say how many bytes they left out, so a model can tell +//! that a line is not whole. + +use crate::text::clip_to_boundary; + +/// How much of a long line Grep shows. +#[derive(Debug, Clone, Copy)] +pub(super) struct Window { + /// The most bytes of a line shown: a longer line is cut. + pub(super) width: usize, + /// How many bytes before the first match a window starts. + pub(super) lead: usize, +} + +impl Default for Window { + /// 1,000 bytes, starting 200 bytes before the first match. + fn default() -> Window { + Window { + width: 1000, + lead: 200, + } + } +} + +/// The marker that says how many bytes of a line are not shown. +fn omitted(bytes: usize) -> String { + format!("[... {bytes} bytes omitted]") +} + +/// The text shown for a matching `line`, whose first match starts at byte +/// `first`. +/// +/// A line within the width is shown whole. A longer line becomes a window +/// that starts up to `lead` bytes before the match and holds at most +/// `width` bytes, with a marker directly before it when the window does +/// not start at the line's start and directly after it when the window +/// stops short of the line's end. The two counts plus the window's length +/// equal the line's length. +pub(super) fn match_text(line: &str, first: usize, window: &Window) -> String { + if line.len() <= window.width { + return line.to_owned(); + } + let start = first.saturating_sub(window.lead); + let shown = clip_to_boundary(line, start, start.saturating_add(window.width)); + // `clip_to_boundary` moves the start up to a character boundary, so the + // bytes before the window are counted from that same boundary. + let mut begin = start.min(line.len()); + while !line.is_char_boundary(begin) { + begin += 1; + } + let after = line.len() - begin - shown.len(); + let mut text = String::new(); + if begin > 0 { + text.push_str(&omitted(begin)); + } + text.push_str(shown); + if after > 0 { + text.push_str(&omitted(after)); + } + text +} + +/// The text shown for a context `line`: the line when it is within the +/// width, and otherwise its first `width` bytes followed by a marker that +/// counts the rest. +pub(super) fn context_text(line: &str, window: &Window) -> String { + if line.len() <= window.width { + return line.to_owned(); + } + let shown = clip_to_boundary(line, 0, window.width); + format!("{shown}{}", omitted(line.len() - shown.len())) +} diff --git a/crates/plugin-std/src/lib.rs b/crates/plugin-std/src/lib.rs index 285d6d215..2d51aec1c 100644 --- a/crates/plugin-std/src/lib.rs +++ b/crates/plugin-std/src/lib.rs @@ -1,6 +1,6 @@ //! `plugin-std` - the Pack that gives a run the standard file tools a -//! coding agent needs: read, write, string replace, delete, glob, and -//! grep. +//! coding agent needs: read, write, string replace, delete, glob, grep, +//! and list directory. //! //! The crate's Plugin, [`install`], inserts the Pack `runvil/std`, whose //! Pack id is `std`, and a run then holds all of its tools unless it @@ -14,6 +14,7 @@ //! | `std/rm` | `Delete` | //! | `std/glob` | `Glob` | //! | `std/grep` | `Grep` | +//! | `std/dir` | `ListDir` | //! //! A script calls a tool by its Lua id, `tools.call("std/read", { path = //! "/notes.md" })`, and offers it to the model with `tools.offer("std")` @@ -30,20 +31,45 @@ //! `schemas/`, embedded at build time. They are Cursor's own //! definitions, so a model that knows Cursor's tools already knows these, //! with the lines that describe behavior this Pack does not have removed. -//! `schemas/EDITS.md` lists every such edit. +//! `schemas/EDITS.md` lists every such edit. ListDir's definition is this +//! crate's own, because Cursor's counterpart is deprecated, and the same +//! file says so. //! //! # This build //! -//! Every tool is a stub: a call with arguments that match the schema -//! answers [`NOT_IMPLEMENTED`], and a call whose arguments do not match -//! fails with an invalid-arguments -//! [`ToolError`](runvil_plugin::ToolError) naming the parameter. The -//! stubs exist so that the crate, the definitions, the ids, the wire -//! names, and the Lua path can be proven before any file logic exists. +//! Read is real. It returns a window of a text file's lines, each with a +//! six-wide line number, and it refuses a directory, a file over 16 MiB, +//! and binary data. A read that stops early ends with a line that names +//! the offset to continue from. File text comes back as untrusted output. +//! +//! ListDir is real. It returns a bullet tree of a directory, listed breadth +//! first, with a trailing `/` on each directory, files and directories in +//! one byte-ordered list, and every entry named `.git` skipped. A link is +//! listed and not entered, and an `ignore_globs` pattern leaves out what +//! it matches at any depth. The tree holds at most 15,000 bytes of lines, +//! and a listing that reaches that says so. Names come back as untrusted +//! output. +//! +//! Grep is real. It searches the files under a path with a Rust regular +//! expression, in path order, and prints each matching line as `N:text` +//! with optional context lines as `N-text`, where `N` is the line number +//! Read shows. Output stops at 15,000 bytes, a call with an `offset` +//! pages through the same result, and a binary file, a link, an over-size +//! file, and an entry that cannot be read are skipped and counted, never +//! a failure. A search yields to the executor now and then, so a cancel +//! or a stop can drop it. File text comes back as untrusted output. +//! +//! Write, StrReplace, Delete, and Glob are still stubs: a call with +//! arguments that match the schema answers [`NOT_IMPLEMENTED`]. For every +//! tool, a call whose arguments do not match fails with an +//! invalid-arguments [`ToolError`](runvil_plugin::ToolError) naming the +//! parameter. The stubs exist so that the crate, the definitions, the +//! ids, the wire names, and the Lua path can be proven before any file +//! logic exists. //! //! # Where the file access goes //! -//! When the tools are real, every filesystem operation goes through the +//! Every filesystem operation a tool performs goes through the //! call's [`Access`](runvil_plugin::Access), never the disk directly. That //! puts every read and write under the run's `ModePolicy`, so a run in //! `Mode::Ask` refuses every mutation without the Pack deciding anything, @@ -69,9 +95,20 @@ //! failure that names the parameter. No other outcome reports the //! arguments, so a model can tell a bad call from a missing //! implementation. +//! - A key the schema does not declare is refused by name, with the +//! parameters listed, by the three real tools, Read, ListDir, and +//! Grep. The stubs let it through. Grep alone accepts `-n` when it is +//! `true`, because Cursor's definition once declared it. mod check; +mod dir; +mod grep; +mod matcher; +mod order; mod pack; +mod read; +mod refusal; mod spec; +mod text; pub use crate::pack::{NOT_IMPLEMENTED, install}; diff --git a/crates/plugin-std/src/matcher.rs b/crates/plugin-std/src/matcher.rs new file mode 100644 index 000000000..7bd8ad625 --- /dev/null +++ b/crates/plugin-std/src/matcher.rs @@ -0,0 +1,224 @@ +//! The glob matcher that ListDir's `ignore_globs` use, and that Grep's +//! `glob` filter uses after it. +//! +//! The grammar is the one `Access::glob` accepts, ported because this +//! crate cannot import the vfs matcher: only `*` and `**` are special, and +//! every other byte is literal and matched case-sensitively. One thing +//! differs. `Access::glob` anchors a pattern with no leading `/` at the +//! access root, and this matcher takes such a pattern at any depth, as if +//! `**/` came first, because it filters paths that are already known. +//! +//! The match is a bounded iterative sweep, with no recursion, so a hostile +//! pattern cannot drive exponential time or blow the stack. + +/// The longest pattern, in bytes, the matcher takes. +pub(crate) const MAX_PATTERN_BYTES: usize = 1024; + +/// Why a pattern was refused. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum PatternError { + /// The pattern is empty. + Empty, + /// The pattern is longer than [`MAX_PATTERN_BYTES`]. + TooLong, + /// The pattern holds a byte below 0x20, or 0x7f. + Control, + /// The pattern holds a backslash, and there is no escape syntax. + Backslash, + /// The pattern holds three or more `*` in a row. + StarRun, + /// A `**` that is not a whole path segment. + PartialDoubleStar, +} + +impl PatternError { + /// The text that says what rule the pattern broke. + pub(crate) fn reason(self) -> &'static str { + match self { + PatternError::Empty => "pattern is empty", + PatternError::TooLong => "pattern is longer than 1024 bytes", + PatternError::Control => "pattern contains a control character", + PatternError::Backslash => "pattern does not support backslash escapes", + PatternError::StarRun => "more than two consecutive '*' are not supported", + PatternError::PartialDoubleStar => "'**' must occupy a whole path segment", + } + } +} + +/// One unit of a valid pattern. +#[derive(Debug, Clone, Copy)] +enum Token { + /// A literal byte that must match exactly. + Literal(u8), + /// `*`: zero or more bytes, none of them `/`. + Star, + /// `**` at the end of the pattern, or alone: zero or more bytes of any + /// kind. + DoubleStar, + /// `**/`: zero or more whole path segments, which is nothing, or a run + /// that ends in `/`. + DoubleStarSlash, +} + +/// A valid pattern, ready to match absolute virtual paths. +#[derive(Debug)] +pub(crate) struct Matcher { + /// The pattern's units, with a leading [`Token::DoubleStarSlash`] when + /// the pattern has no leading `/`. + tokens: Vec, +} + +impl Matcher { + /// Compiles `pattern`. + /// + /// A pattern with a leading `/` is a full path from the root. Any + /// other pattern matches at any depth, as if `**/` came first. + /// + /// # Errors + /// Returns the rule the pattern breaks: empty, longer than + /// [`MAX_PATTERN_BYTES`], holding a control byte or a backslash, + /// holding more than two `*` in a row, or holding a `**` that is not a + /// whole path segment. The rules are checked in that order. + pub(crate) fn new(pattern: &str) -> Result { + validate(pattern)?; + let mut tokens = Vec::with_capacity(pattern.len() + 1); + if !pattern.starts_with('/') { + tokens.push(Token::DoubleStarSlash); + } + tokenize(pattern.as_bytes(), &mut tokens); + Ok(Matcher { tokens }) + } + + /// Whether the whole of `path`, an absolute virtual path with a leading + /// `/`, matches. + /// + /// The sweep tracks which prefixes of `path` the tokens read so far can + /// consume, so it takes `O(tokens * path length)` time. + pub(crate) fn is_match(&self, path: &str) -> bool { + let text = path.as_bytes(); + let len = text.len(); + // `reachable[j]` is true when the tokens read so far match exactly + // `text[..j]`. + let mut reachable = vec![false; len + 1]; + if let Some(start) = reachable.first_mut() { + *start = true; + } + let mut next = vec![false; len + 1]; + for token in &self.tokens { + next.fill(false); + match *token { + Token::Literal(byte) => { + for (j, here) in reachable.iter().take(len).enumerate() { + if *here + && text.get(j) == Some(&byte) + && let Some(after) = next.get_mut(j + 1) + { + *after = true; + } + } + } + Token::Star => { + // Zero or more bytes that are not `/`: carry + // reachability forward across each such byte. + let mut carry = false; + for (j, (slot, here)) in next.iter_mut().zip(&reachable).enumerate() { + let reached = *here || carry; + *slot = reached; + carry = reached && text.get(j).is_some_and(|byte| *byte != b'/'); + } + } + Token::DoubleStar => { + // Zero or more bytes of any kind: once a position is + // reachable, every later one is too. + let mut seen = false; + for (slot, here) in next.iter_mut().zip(&reachable) { + seen |= *here; + *slot = seen; + } + } + Token::DoubleStarSlash => { + // Nothing, or a run that ends in `/`. + let mut seen = false; + for (j, (slot, here)) in next.iter_mut().zip(&reachable).enumerate() { + let after_slash = j + .checked_sub(1) + .and_then(|before| text.get(before)) + .is_some_and(|byte| *byte == b'/'); + *slot = *here || (seen && after_slash); + seen |= *here; + } + } + } + std::mem::swap(&mut reachable, &mut next); + } + reachable.last().copied().unwrap_or(false) + } +} + +/// Checks `pattern` against the rules, in a fixed order: empty, length, +/// control bytes, backslash, then the `*` grammar. +fn validate(pattern: &str) -> Result<(), PatternError> { + if pattern.is_empty() { + return Err(PatternError::Empty); + } + if pattern.len() > MAX_PATTERN_BYTES { + return Err(PatternError::TooLong); + } + if pattern.bytes().any(|byte| byte < 0x20 || byte == 0x7f) { + return Err(PatternError::Control); + } + if pattern.contains('\\') { + return Err(PatternError::Backslash); + } + let bytes = pattern.as_bytes(); + let mut index = 0; + while index < bytes.len() { + if bytes.get(index) != Some(&b'*') { + index += 1; + continue; + } + let run_start = index; + while bytes.get(index) == Some(&b'*') { + index += 1; + } + match index - run_start { + 1 => {} + 2 => { + let before_ok = run_start + .checked_sub(1) + .is_none_or(|before| bytes.get(before) == Some(&b'/')); + let after_ok = bytes.get(index).is_none_or(|byte| *byte == b'/'); + if !(before_ok && after_ok) { + return Err(PatternError::PartialDoubleStar); + } + } + _ => return Err(PatternError::StarRun), + } + } + Ok(()) +} + +/// Turns the valid pattern `pattern` into tokens, pushed onto `tokens`. +fn tokenize(pattern: &[u8], tokens: &mut Vec) { + let mut index = 0; + while let Some(byte) = pattern.get(index) { + if *byte != b'*' { + tokens.push(Token::Literal(*byte)); + index += 1; + } else if pattern.get(index + 1) == Some(&b'*') { + if pattern.get(index + 2) == Some(&b'/') { + tokens.push(Token::DoubleStarSlash); + index += 3; + } else { + tokens.push(Token::DoubleStar); + index += 2; + } + } else { + tokens.push(Token::Star); + index += 1; + } + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/plugin-std/src/matcher/tests.rs b/crates/plugin-std/src/matcher/tests.rs new file mode 100644 index 000000000..cffce8daa --- /dev/null +++ b/crates/plugin-std/src/matcher/tests.rs @@ -0,0 +1,275 @@ +//! Unit tests for the glob matcher: each grammar rule, the refusals with +//! their reason texts, and the cases it shares with `Access::glob`, which +//! are pinned against that function's own tables. + +use runvil_plugin::ToolId; +use runvil_plugin::testing::TestCall; + +use super::{MAX_PATTERN_BYTES, Matcher, PatternError}; + +/// A matcher for `pattern`, which must be valid. +fn matcher(pattern: &str) -> Matcher { + Matcher::new(pattern).expect("the pattern is valid") +} + +/// Whether the valid `pattern` matches the absolute virtual `path`. +fn matches(pattern: &str, path: &str) -> bool { + matcher(pattern).is_match(path) +} + +/// The reason text of the refusal of `pattern`. +fn reason_of(pattern: &str) -> &'static str { + match Matcher::new(pattern) { + Ok(_) => panic!("{pattern:?} is refused"), + Err(error) => error.reason(), + } +} + +#[test] +fn a_star_matches_zero_or_more_bytes_inside_one_segment() { + assert!(matches("/a/*.rs", "/a/b.rs")); + assert!(matches("/a/*.rs", "/a/.rs")); + assert!(matches("/a/b*", "/a/b")); + assert!(!matches("/a/*.rs", "/a/x/b.rs")); + assert!(!matches("/a/*", "/a/x/y")); +} + +#[test] +fn a_star_matches_dotfiles() { + assert!(matches("/*", "/.env")); + assert!(matches("/src/*", "/src/.hidden")); + assert!(matches("*.log", "/.debug.log")); + assert!(matches("/.*", "/.gitignore")); +} + +#[test] +fn a_double_star_segment_matches_zero_or_more_whole_segments() { + assert!(matches("/a/**/z", "/a/z")); + assert!(matches("/a/**/z", "/a/b/z")); + assert!(matches("/a/**/z", "/a/b/c/z")); + assert!(!matches("/a/**/z", "/a/bz")); + assert!(!matches("/a/**/z", "/b/z")); +} + +#[test] +fn a_trailing_double_star_matches_everything_below() { + assert!(matches("/a/**", "/a/b")); + assert!(matches("/a/**", "/a/b/c/d")); + assert!(!matches("/a/**", "/b/c")); +} + +#[test] +fn every_byte_but_the_star_is_literal() { + assert!(matches("/a?", "/a?")); + assert!(!matches("/a?", "/ab")); + assert!(matches("/[ab]", "/[ab]")); + assert!(!matches("/[ab]", "/a")); + assert!(matches("/{a,b}", "/{a,b}")); + assert!(!matches("/{a,b}", "/a")); + assert!(matches("/!x", "/!x")); + assert!(matches("/a.rs", "/a.rs")); + assert!(!matches("/a.rs", "/aXrs")); +} + +#[test] +fn matching_is_case_sensitive() { + assert!(matches("/README.md", "/README.md")); + assert!(!matches("/README.md", "/readme.md")); + assert!(!matches("*.TXT", "/a.txt")); +} + +#[test] +fn a_pattern_without_a_leading_slash_matches_at_any_depth() { + assert!(matches("node_modules", "/node_modules")); + assert!(matches("node_modules", "/proj/node_modules")); + assert!(matches("node_modules", "/a/b/c/node_modules")); + assert!(matches("*.log", "/x.log")); + assert!(matches("*.log", "/a/b/x.log")); + assert!(matches("src/*.rs", "/proj/src/a.rs")); +} + +#[test] +fn an_any_depth_pattern_still_matches_a_whole_segment_run() { + assert!(!matches("node_modules", "/proj/my_node_modules")); + assert!(!matches("node_modules", "/proj/node_modules_old")); + assert!(!matches("src/*.rs", "/proj/mysrc/a.rs")); + assert!(!matches("*.log", "/a.log/b")); +} + +#[test] +fn a_pattern_with_a_leading_slash_is_a_full_path_from_the_root() { + assert!(matches("/build", "/build")); + assert!(!matches("/build", "/proj/build")); + assert!(matches("/proj/build", "/proj/build")); + assert!(!matches("/proj/build", "/other/proj/build")); + assert!(matches("/", "/")); + assert!(!matches("/", "/a")); +} + +#[test] +fn a_leading_double_star_matches_with_or_without_the_slash_form() { + assert!(matches("**/*.md", "/notes/today.md")); + assert!(matches("**/*.md", "/today.md")); + assert!(matches("/**/*.md", "/today.md")); + assert!(matches("/**/*.md", "/a/b/today.md")); +} + +#[test] +fn every_refusal_has_its_reason_text() { + assert_eq!(reason_of(""), "pattern is empty"); + assert_eq!( + reason_of(&"a".repeat(MAX_PATTERN_BYTES + 1)), + "pattern is longer than 1024 bytes" + ); + assert_eq!(reason_of("a\u{0}b"), "pattern contains a control character"); + assert_eq!( + reason_of("a\u{1f}b"), + "pattern contains a control character" + ); + assert_eq!( + reason_of("a\u{7f}b"), + "pattern contains a control character" + ); + assert_eq!( + reason_of("a\\b"), + "pattern does not support backslash escapes" + ); + assert_eq!( + reason_of("a***"), + "more than two consecutive '*' are not supported" + ); + assert_eq!( + reason_of("a/***/b"), + "more than two consecutive '*' are not supported" + ); + assert_eq!(reason_of("a/**b"), "'**' must occupy a whole path segment"); + assert_eq!(reason_of("a**"), "'**' must occupy a whole path segment"); + assert_eq!(reason_of("**a/b"), "'**' must occupy a whole path segment"); +} + +#[test] +fn the_longest_allowed_pattern_is_1024_bytes() { + assert_eq!(MAX_PATTERN_BYTES, 1024); + assert!(Matcher::new(&"a".repeat(1024)).is_ok()); + assert_eq!( + Matcher::new(&"a".repeat(1025)).err(), + Some(PatternError::TooLong) + ); +} + +#[test] +fn a_refusal_is_checked_in_a_fixed_order() { + // Too long wins over a control byte, and a control byte over a + // backslash, so the same bad pattern always hears the same reason. + let long_and_control = format!("{}\u{0}", "a".repeat(MAX_PATTERN_BYTES)); + assert_eq!( + reason_of(&long_and_control), + "pattern is longer than 1024 bytes" + ); + assert_eq!(reason_of("\u{0}\\"), "pattern contains a control character"); + assert_eq!( + reason_of("\\***"), + "pattern does not support backslash escapes" + ); +} + +#[test] +fn the_validation_table_matches_the_one_access_glob_pins() { + // The cases `glob.rs` pins in the vfs crate: the over-long, control, + // backslash, and wildcard refusals, and one pattern that is fine. + let over_long = "a".repeat(MAX_PATTERN_BYTES + 1); + for (pattern, refused) in [ + (over_long.as_str(), true), + ("a\u{0}b", true), + ("a\\b", true), + ("a/***/b", true), + ("a/**b", true), + ("ok/*.txt", false), + ] { + assert_eq!( + Matcher::new(pattern).is_err(), + refused, + "{pattern:?} is refused: {refused}" + ); + } +} + +#[test] +fn the_match_table_is_the_one_access_glob_pins_when_anchored_with_a_leading_slash() { + // The cases `glob.rs` pins, run anchored: each pattern and each path + // gets a leading `/`, so a pattern is a full path from the root, as it + // is for `Access::glob`. + let paths = [ + "a.txt", + "src/a.rs", + "src/b.rs", + "src/deep/c.rs", + "src/deep/deeper/d.rs", + "notes/today.md", + ]; + for (pattern, expected) in [ + ("*.txt", [true, false, false, false, false, false]), + ("src/*.rs", [false, true, true, false, false, false]), + ("src/**/*.rs", [false, true, true, true, true, false]), + ("**/*.md", [false, false, false, false, false, true]), + ("src/**", [false, true, true, true, true, false]), + ("no*match", [false, false, false, false, false, false]), + ] { + let matcher = matcher(&format!("/{pattern}")); + for (path, expected) in paths.iter().zip(expected) { + assert_eq!( + matcher.is_match(&format!("/{path}")), + expected, + "pattern /{pattern} on /{path}" + ); + } + } +} + +#[test] +fn an_unanchored_pattern_is_the_one_way_matching_differs_from_access_glob() { + // `Access::glob("*.txt")` is anchored at the access root, so `*` never + // crosses a `/` and only the top-level file comes back. The matcher + // takes a pattern with no leading `/` at any depth. + let call = TestCall::new(ToolId::parse("std/dir").expect("a two-segment tool id parses")); + let access = call.context().access(); + access.write("/a.txt", b"").expect("the file is written"); + access + .write("/x/y/a.txt", b"") + .expect("the file is written"); + let globbed = access.glob("*.txt").expect("the pattern is valid"); + assert_eq!(globbed, ["a.txt"]); + let any_depth = matcher("*.txt"); + assert!(any_depth.is_match("/a.txt")); + assert!(any_depth.is_match("/x/y/a.txt")); +} + +#[test] +fn a_long_text_does_not_blow_the_stack_or_the_clock() { + // The match is an iterative sweep, so a deep path and a pattern full + // of stars finish at once. + let deep = format!("/{}", "d/".repeat(5_000)); + assert!(matches("**/**/**/x", &format!("{deep}x"))); + assert!(!matches("**/**/**/x", &format!("{deep}y"))); + assert!(matches("*a*a*a*a*b", &format!("/{}b", "a".repeat(2_000)))); + assert!(!matches("*a*a*a*a*b", &format!("/{}", "a".repeat(2_000)))); +} + +#[test] +fn the_reason_texts_are_distinct() { + let reasons = [ + PatternError::Empty, + PatternError::TooLong, + PatternError::Control, + PatternError::Backslash, + PatternError::StarRun, + PatternError::PartialDoubleStar, + ] + .map(PatternError::reason); + for (index, reason) in reasons.iter().enumerate() { + assert!( + !reasons.iter().skip(index + 1).any(|other| other == reason), + "{reason} appears twice" + ); + } +} diff --git a/crates/plugin-std/src/order.rs b/crates/plugin-std/src/order.rs new file mode 100644 index 000000000..0d3c85fdb --- /dev/null +++ b/crates/plugin-std/src/order.rs @@ -0,0 +1,32 @@ +//! The order and the one name the directory-walking tools share. +//! +//! ListDir and Grep both walk a directory tree, and the order they visit +//! entries in is a promise to the model: it is the order `Access::glob` +//! gives, so a walk and a glob over the same tree list the same paths in +//! the same sequence. Both tools take the order and the skipped name from +//! here, so the two cannot drift apart. + +use runvil_plugin::{Entry, FileType}; + +/// The name of the entry a walk skips, a file or a directory, at any +/// depth. +pub(crate) const GIT: &str = ".git"; + +/// `entries` in the order a walk visits them: by name, with `/` added to +/// a directory's name. +/// +/// `Access::list` promises no order. This is the order `Access::glob` +/// gives, which sorts full path strings bytewise. +pub(crate) fn sorted(mut entries: Vec) -> Vec { + entries.sort_by_cached_key(|entry| { + if matches!(entry.stat.file_type, FileType::Directory) { + format!("{}/", entry.name) + } else { + entry.name.clone() + } + }); + entries +} + +#[cfg(test)] +mod tests; diff --git a/crates/plugin-std/src/order/tests.rs b/crates/plugin-std/src/order/tests.rs new file mode 100644 index 000000000..1eb5b2fad --- /dev/null +++ b/crates/plugin-std/src/order/tests.rs @@ -0,0 +1,56 @@ +//! Unit tests for the shared walk order: the one sequence ListDir and +//! Grep both visit entries in. + +use runvil_plugin::{Entry, FileType, Stat}; + +use super::{GIT, sorted}; + +/// An entry called `name` of the given kind. +fn entry(name: &str, file_type: FileType) -> Entry { + Entry::new(name, Stat::new(file_type, 0)) +} + +/// The names of `entries`, in order. +fn names(entries: &[Entry]) -> Vec<&str> { + entries.iter().map(|entry| entry.name.as_str()).collect() +} + +#[test] +fn a_directory_sorts_as_its_name_with_a_slash_added() { + let entries = vec![ + entry("b", FileType::File), + entry("a.txt", FileType::File), + entry("a", FileType::Directory), + entry("a-b", FileType::File), + entry("B", FileType::File), + ]; + // Full paths `/B`, `/a-b`, `/a.txt`, `/a/x`, `/b` sort bytewise in + // this order: `-` is below `.`, which is below `/`. + assert_eq!(names(&sorted(entries)), ["B", "a-b", "a.txt", "a", "b"]); +} + +#[test] +fn a_directory_and_a_file_that_share_a_prefix_follow_the_path_order() { + let entries = vec![ + entry("x.rs", FileType::File), + entry("x", FileType::Directory), + entry("x-y", FileType::File), + ]; + assert_eq!(names(&sorted(entries)), ["x-y", "x.rs", "x"]); +} + +#[test] +fn the_order_does_not_depend_on_the_order_listed() { + let forward = vec![ + entry("a", FileType::Directory), + entry("a.txt", FileType::File), + entry("b", FileType::File), + ]; + let backward: Vec = forward.iter().rev().cloned().collect(); + assert_eq!(names(&sorted(forward)), names(&sorted(backward))); +} + +#[test] +fn the_skipped_name_is_dot_git() { + assert_eq!(GIT, ".git"); +} diff --git a/crates/plugin-std/src/pack-tests.rs b/crates/plugin-std/src/pack-tests.rs index e436e5929..36ef7a619 100644 --- a/crates/plugin-std/src/pack-tests.rs +++ b/crates/plugin-std/src/pack-tests.rs @@ -47,11 +47,14 @@ fn the_stub_says_what_is_missing_before_it_says_anything_else() { } #[test] -fn the_specs_are_the_six_the_crate_names_in_order() { +fn the_specs_are_the_seven_the_crate_names_in_order() { let segments: Vec<&str> = spec::all() .expect("every embedded definition parses") .iter() .map(|definition| definition.segment) .collect(); - assert_eq!(segments, ["read", "write", "replace", "rm", "glob", "grep"]); + assert_eq!( + segments, + ["read", "write", "replace", "rm", "glob", "grep", "dir"] + ); } diff --git a/crates/plugin-std/src/pack.rs b/crates/plugin-std/src/pack.rs index c70f3b2e6..47dce1e51 100644 --- a/crates/plugin-std/src/pack.rs +++ b/crates/plugin-std/src/pack.rs @@ -1,6 +1,6 @@ //! The crate's Plugin, [`install`], the `std` Pack's `PackInfo`, the -//! `construct` that builds its one shared object, and the stub every tool -//! answers with in this build. +//! `construct` that builds its one shared object, and the stub that every +//! tool but Read, ListDir, and Grep still answers with in this build. use std::sync::Arc; @@ -12,7 +12,9 @@ use serde_json::{Value, json}; use crate::spec::{self, Definition}; -/// What a call whose arguments match the schema answers in this build. +/// What a call to a tool that is still a stub answers in this build, when +/// its arguments match the schema. Read, ListDir, and Grep are real and +/// never answer this. /// /// The text says the implementation is missing rather than that the call /// was wrong, because a model that reads it should stop calling the tool. @@ -118,7 +120,12 @@ impl Pack for Std { return Err(ToolError::message(format!("std has no tool named {name}"))); }; crate::check::check(&definition.wire_name, &definition.parameters, &args)?; - Ok(ToolOutput::trusted(stub(&definition.wire_name))) + match definition.segment { + "read" => crate::read::call(cx.access(), definition, &args), + "dir" => crate::dir::call(cx.access(), definition, &args), + "grep" => crate::grep::call(cx.access(), definition, &args).await, + _ => Ok(ToolOutput::trusted(stub(&definition.wire_name))), + } }) } } diff --git a/crates/plugin-std/src/read.rs b/crates/plugin-std/src/read.rs new file mode 100644 index 000000000..789759242 --- /dev/null +++ b/crates/plugin-std/src/read.rs @@ -0,0 +1,100 @@ +//! The `Read` tool: a window of a text file's lines, numbered. +//! +//! Read opens the path the call names through the call's [`Access`], so +//! the run's mode and policy apply. It refuses a directory, a special +//! file, a file over 16 MiB, and binary data, and it never truncates a +//! file: a read that does not reach the last line says which offset +//! continues it. File text is returned as untrusted output. + +mod args; +mod render; +mod window; + +#[cfg(test)] +mod tests; + +use runvil_plugin::{Access, FileType, ToolError, ToolOutput, VfsError}; +use serde_json::Value; + +use crate::check::plural; +use crate::refusal::{neither_file_nor_directory, refusal}; +use crate::spec::Definition; +use crate::text::{MAX_FILE_BYTES, decode, sniff_binary, split_lines}; + +use self::args::Request; + +/// The most bytes of numbered lines one read returns: each line's prefix, +/// its text, and its newline count, and the notes and the trailer do not. +pub(crate) const BYTE_CAP: usize = 100_000; + +/// Runs one `Read` call whose arguments already passed the shared schema +/// check. +/// +/// # Errors +/// Returns an invalid-arguments [`ToolError`] naming the parameter for a +/// bad argument, and a message error for any other failure, worded for a +/// model to read. +pub(crate) fn call( + access: &Access, + definition: &Definition, + args: &Value, +) -> Result { + let request = args::parse(definition, args)?; + read_file(access, &definition.wire_name, &request, BYTE_CAP) +} + +/// Reads the file `request` names, in the window it asks for, with at +/// most `cap` bytes of numbered lines. `tool` is the name the model +/// called the tool by. +fn read_file( + access: &Access, + tool: &str, + request: &Request<'_>, + cap: usize, +) -> Result { + let path = request.path; + let failed = |error: VfsError| ToolError::message(refusal(tool, path, &error)); + let stat = access.stat(path).map_err(failed)?; + check_type(path, stat.file_type).map_err(ToolError::message)?; + let bytes = access + .read_range(path, 0, MAX_FILE_BYTES + 1) + .map_err(failed)?; + if u64::try_from(bytes.len()).unwrap_or(u64::MAX) > MAX_FILE_BYTES { + return Err(ToolError::message(format!( + "File is too large to read: {path} is over the limit of {MAX_FILE_BYTES} bytes." + ))); + } + if sniff_binary(&bytes) { + return Err(ToolError::message(format!( + "File content is binary data, not text: {path}" + ))); + } + let decoded = decode(&bytes); + let lines = split_lines(&decoded.text); + if lines.is_empty() { + return Ok(ToolOutput::untrusted("File is empty.")); + } + let Some(window) = window::window(lines.len(), request.offset, request.limit) else { + return Err(ToolError::message(format!( + "Offset {} is beyond the end of {path}, which has {}.", + request.offset.unwrap_or_default(), + plural(lines.len(), "line") + ))); + }; + let notes = render::notes(decoded.utf16, decoded.invalid); + Ok(ToolOutput::untrusted(render::render( + &lines, &window, ¬es, cap, + ))) +} + +/// Decides from the kind of node at `path` whether Read goes on: a file +/// or a link does, and anything else gets the sentence that says why not. +fn check_type(path: &str, file_type: FileType) -> Result<(), String> { + match file_type { + FileType::File | FileType::Symlink => Ok(()), + FileType::Directory => Err(format!("Path is a directory, not a file: {path}")), + // A pipe, a socket, or a device, and any kind added later: none + // of them is text Read can window. + _ => Err(neither_file_nor_directory(path)), + } +} diff --git a/crates/plugin-std/src/read/args.rs b/crates/plugin-std/src/read/args.rs new file mode 100644 index 000000000..94721399d --- /dev/null +++ b/crates/plugin-std/src/read/args.rs @@ -0,0 +1,102 @@ +//! Read's arguments: what the call asks for, and what is refused before +//! any file is touched. + +use runvil_plugin::ToolError; +use serde_json::Value; + +use crate::check::{invalid_arguments, refuse_blank, refuse_unknown}; +use crate::spec::Definition; + +/// What a call asks Read for. +#[derive(Debug)] +pub(super) struct Request<'a> { + /// The path as the model wrote it. + pub(super) path: &'a str, + /// The line to start at: `0` and an absent offset read as 1, and a + /// negative one counts back from the end. + pub(super) offset: Option, + /// The most lines to read, at least 1. + pub(super) limit: Option, +} + +/// Reads the call's arguments after the shared schema check. +/// +/// The order is fixed: an undeclared key, then a blank path, then the +/// offset, then the limit, so a call that breaks two rules always hears +/// about the same one. +/// +/// # Errors +/// Returns an invalid-arguments [`ToolError`] naming the parameter. +pub(super) fn parse<'a>( + definition: &Definition, + args: &'a Value, +) -> Result, ToolError> { + let tool = definition.wire_name.as_str(); + refuse_unknown(tool, &definition.parameters, args)?; + let path = args.get("path").and_then(Value::as_str).unwrap_or_default(); + refuse_blank(tool, "path", path)?; + let offset = integer(tool, "offset", args)?.map(|(offset, _)| offset); + let limit = match integer(tool, "limit", args)? { + Some((limit, given)) if limit < 1 => { + return Err(invalid_arguments( + tool, + format!("`limit` must be at least 1, and the call passed {given}"), + )); + } + Some((limit, _)) => Some(usize::try_from(limit).unwrap_or(usize::MAX)), + None => None, + }; + Ok(Request { + path, + offset, + limit, + }) +} + +/// The integer under `name` in `args` with the value as the call wrote +/// it, or `None` when the call left it out or passed `null`. +/// +/// The shared check has already refused anything that is not a whole +/// number, so a value that still has no `i64` is a whole number past the +/// 64-bit range, which is refused here. +fn integer<'a>( + tool: &str, + name: &str, + args: &'a Value, +) -> Result, ToolError> { + let Some(value) = args.get(name).filter(|value| !value.is_null()) else { + return Ok(None); + }; + whole_number(value) + .map(|number| Some((number, value))) + .ok_or_else(|| { + invalid_arguments( + tool, + format!( + "`{name}` must be a JSON integer that fits in 64 bits, \ + and the call passed {value}" + ), + ) + }) +} + +/// The `i64` a JSON number holds, counting a whole number written with a +/// fraction part such as `3.0`, which the shared check accepts as an +/// integer. `None` for a number outside the `i64` range. +fn whole_number(value: &Value) -> Option { + if let Some(integer) = value.as_i64() { + return Some(integer); + } + let float = value.as_f64()?; + if float.fract() != 0.0 || !(-I64_BOUND..I64_BOUND).contains(&float) { + return None; + } + #[expect( + clippy::cast_possible_truncation, + reason = "the value is whole and inside the i64 range, so the cast is exact" + )] + Some(float as i64) +} + +/// 2^63, the first `f64` above the `i64` range (`-2^63` is `i64::MIN`). +const I64_BOUND: f64 = 9_223_372_036_854_775_808.0; diff --git a/crates/plugin-std/src/read/render.rs b/crates/plugin-std/src/read/render.rs new file mode 100644 index 000000000..87ba6eb1d --- /dev/null +++ b/crates/plugin-std/src/read/render.rs @@ -0,0 +1,118 @@ +//! The render: turns a window of lines into the text Read answers with. +//! +//! Each line prints as `{number:>6}|{text}`: a six-wide right-aligned +//! number, a pipe, and no space. The number is always on and widens past +//! six digits, and it is not part of the line. The numbered lines are +//! followed by the notes and then the trailer, each on its own line, with +//! no newline after the last one. + +use crate::check::plural; +use crate::text::clip_to_boundary; + +use super::window::Window; + +/// The numbered prefix of line `number`. +fn prefix(number: usize) -> String { + format!("{number:>6}|") +} + +/// The text of the `window` over `lines`, with `notes` and a trailer when +/// the read stops early. +/// +/// The numbered lines hold at most `cap` bytes, counting each prefix, the +/// text, and its newline, and not the notes or the trailer. A line that +/// would pass the cap is left out, and the trailer names the offset that +/// continues. When the window's first line alone passes the cap, it is +/// cut to fit and printed alone, and the trailer says so: this is the only +/// cut inside a line. A read that reaches the last line prints no trailer. +/// +/// `window` lies inside `lines`. +pub(super) fn render(lines: &[&str], window: &Window, notes: &[String], cap: usize) -> String { + let total = lines.len(); + let mut body = String::new(); + let mut used = 0usize; + let mut shown = 0usize; + for number in window.first..=window.last { + let Some(text) = lines.get(number - 1) else { + break; + }; + let prefix = prefix(number); + let cost = prefix.len() + text.len() + 1; + if used + cost > cap { + break; + } + body.push_str(&prefix); + body.push_str(text); + body.push('\n'); + used += cost; + shown = number; + } + let trailer = if shown == 0 { + Some(cut_first_line(lines, window.first, cap, &mut body)) + } else if shown < window.last { + Some(format!( + "[Showing lines {first}-{shown} of {total} ({cap}-byte limit). \ + Use offset={next} to continue.]", + first = window.first, + next = shown + 1, + )) + } else if window.last < total { + Some(format!( + "[Showing lines {first}-{last} of {total}. Use offset={next} to continue.]", + first = window.first, + last = window.last, + next = window.last + 1, + )) + } else { + None + }; + let tail: Vec<&str> = notes + .iter() + .map(String::as_str) + .chain(trailer.as_deref()) + .collect(); + body.push_str(&tail.join("\n")); + body +} + +/// Prints the first line of the window, number `number`, cut to fit +/// `cap` bytes, into `body`, and returns the trailer that says it was +/// cut. +fn cut_first_line(lines: &[&str], number: usize, cap: usize, body: &mut String) -> String { + let text = lines.get(number - 1).copied().unwrap_or_default(); + let prefix = prefix(number); + let allowance = cap.saturating_sub(prefix.len() + 1); + let kept = clip_to_boundary(text, 0, allowance); + body.push_str(&prefix); + body.push_str(kept); + body.push('\n'); + let next = if number < lines.len() { + format!(" Use offset={} to continue.", number + 1) + } else { + String::new() + }; + format!( + "[Line {number} is {bytes} bytes and is cut after {cut} bytes. \ + No parameter reads the rest of a line.{next}]", + bytes = text.len(), + cut = kept.len(), + ) +} + +/// The notes for a file that was decoded, in the order they print: the +/// UTF-16 note first, then the note that counts replaced bytes. +pub(super) fn notes(utf16: Option<&str>, invalid: usize) -> Vec { + let mut notes = Vec::new(); + if let Some(name) = utf16 { + notes.push(format!( + "[Note: the file is {name} and was decoded to text.]" + )); + } + if invalid > 0 { + notes.push(format!( + "[Note: {} in this file were shown as U+FFFD.]", + plural(invalid, "invalid UTF-8 sequence") + )); + } + notes +} diff --git a/crates/plugin-std/src/read/tests.rs b/crates/plugin-std/src/read/tests.rs new file mode 100644 index 000000000..2500c0e73 --- /dev/null +++ b/crates/plugin-std/src/read/tests.rs @@ -0,0 +1,99 @@ +//! Unit tests for the `Read` tool. This file holds the helpers the topic +//! files share: a call over a fresh in-memory filesystem, a seeded file, +//! and the two ways to drive Read, through its arguments and through a +//! lowered byte cap. + +mod arguments; +mod failures; +mod paging; +mod render; +mod window; + +use runvil_plugin::testing::TestCall; +use runvil_plugin::{OutputTrust, ToolError, ToolId, ToolOutput}; +use serde_json::Value; + +use super::args::Request; +use crate::spec::{self, Definition}; + +/// A call of `std/read` over a fresh in-memory filesystem. +fn new_call() -> TestCall { + TestCall::new(ToolId::parse("std/read").expect("a two-segment tool id parses")) +} + +/// Writes `bytes` to `path` through the call's own access. +fn seed(call: &TestCall, path: &str, bytes: &[u8]) { + call.context() + .access() + .write(path, bytes) + .expect("a seeded file is written"); +} + +/// A call with `bytes` already at `path`. +fn seeded(path: &str, bytes: &[u8]) -> TestCall { + let call = new_call(); + seed(&call, path, bytes); + call +} + +/// The `read` definition the Pack builds the tool from. +fn definition() -> &'static Definition { + spec::all() + .expect("every embedded definition parses") + .iter() + .find(|definition| definition.segment == "read") + .expect("the read definition exists") +} + +/// Calls Read with `args` as the Pack does: the schema check first, then +/// the tool. +fn read(call: &TestCall, args: &Value) -> Result { + let definition = definition(); + crate::check::check(&definition.wire_name, &definition.parameters, args)?; + super::call(call.context().access(), definition, args) +} + +/// Reads `path` with the window `offset` and `limit` under the byte cap +/// `cap`. +fn read_capped( + call: &TestCall, + path: &str, + offset: Option, + limit: Option, + cap: usize, +) -> Result { + let request = Request { + path, + offset, + limit, + }; + super::read_file(call.context().access(), "Read", &request, cap) +} + +/// The text of a successful read, which must be untrusted output. +fn text_of(result: Result) -> String { + let output = result.expect("the read succeeds"); + assert_eq!(output.trust(), OutputTrust::Untrusted, "{}", output.text()); + output.text().to_owned() +} + +/// The message of a failed read. +fn failure_of(result: Result) -> String { + match result { + Ok(output) => panic!("the read fails, but it answered {:?}", output.text()), + Err(error) => error.to_string(), + } +} + +/// A file of `count` lines, `line 1` to `line {count}`, with or without a +/// final newline. +fn numbered_file(count: usize, final_newline: bool) -> String { + let mut text = (1..=count) + .map(|n| format!("line {n}")) + .collect::>() + .join("\n"); + if final_newline && count > 0 { + text.push('\n'); + } + text +} diff --git a/crates/plugin-std/src/read/tests/arguments.rs b/crates/plugin-std/src/read/tests/arguments.rs new file mode 100644 index 000000000..7918da741 --- /dev/null +++ b/crates/plugin-std/src/read/tests/arguments.rs @@ -0,0 +1,216 @@ +//! The argument rules: what Read refuses before it touches a file, and in +//! what order. + +use runvil_plugin::ToolErrorKind; +use serde_json::{Value, json}; + +use super::{new_call, numbered_file, read, seeded, text_of}; + +/// The error kind and message of a refused call over an empty filesystem. +fn refused(args: &Value) -> (ToolErrorKind, String) { + let call = new_call(); + match read(&call, args) { + Ok(output) => panic!("{args} is refused, but it answered {:?}", output.text()), + Err(error) => (error.kind(), error.to_string()), + } +} + +#[test] +fn a_limit_below_one_is_refused_with_the_value_as_given() { + for (limit, passed) in [(json!(0), "0"), (json!(-3), "-3")] { + let (kind, message) = refused(&json!({ "path": "/a.md", "limit": limit })); + assert_eq!(kind, ToolErrorKind::InvalidArguments); + assert_eq!( + message, + format!("std_Read: `limit` must be at least 1, and the call passed {passed}") + ); + } +} + +#[test] +fn a_string_offset_is_refused_and_not_coerced() { + let (kind, message) = refused(&json!({ "path": "/a.md", "offset": "3" })); + assert_eq!(kind, ToolErrorKind::InvalidArguments); + assert_eq!( + message, + "std_Read: `offset` must be a JSON integer, and the call passed \"3\"" + ); +} + +#[test] +fn a_string_limit_and_a_fractional_offset_are_refused() { + let (_, message) = refused(&json!({ "path": "/a.md", "limit": "3" })); + assert_eq!( + message, + "std_Read: `limit` must be a JSON integer, and the call passed \"3\"" + ); + let (_, message) = refused(&json!({ "path": "/a.md", "offset": 1.5 })); + assert_eq!( + message, + "std_Read: `offset` must be a JSON integer, and the call passed 1.5" + ); +} + +#[test] +fn an_integer_that_does_not_fit_in_64_bits_is_refused() { + for name in ["offset", "limit"] { + let mut args = json!({ "path": "/a.md" }); + args[name] = json!(u64::MAX); + let (kind, message) = refused(&args); + assert_eq!(kind, ToolErrorKind::InvalidArguments); + assert_eq!( + message, + format!( + "std_Read: `{name}` must be a JSON integer that fits in 64 bits, \ + and the call passed 18446744073709551615" + ) + ); + } +} + +#[test] +fn a_whole_number_written_as_a_float_reads_as_that_integer() { + let call = seeded("/a.md", numbered_file(10, true).as_bytes()); + for (written, plain) in [ + (json!({ "offset": 3.0 }), json!({ "offset": 3 })), + (json!({ "offset": -2.0 }), json!({ "offset": -2 })), + (json!({ "offset": 0.0 }), json!({ "offset": 0 })), + (json!({ "limit": 2.0 }), json!({ "limit": 2 })), + ( + json!({ "offset": 4.0, "limit": 3.0 }), + json!({ "offset": 4, "limit": 3 }), + ), + ( + json!({ "offset": -9_223_372_036_854_775_808.0_f64 }), + json!({ "offset": i64::MIN }), + ), + ] { + let mut written_args = written.clone(); + written_args["path"] = json!("/a.md"); + let mut plain_args = plain; + plain_args["path"] = json!("/a.md"); + assert_eq!( + text_of(read(&call, &written_args)), + text_of(read(&call, &plain_args)), + "{written}" + ); + } +} + +#[test] +fn a_whole_number_float_past_the_64_bit_range_is_refused() { + for (name, passed) in [ + ("offset", json!(1e30)), + ("limit", json!(1e30)), + ("offset", json!(9_223_372_036_854_775_808.0_f64)), + ("offset", json!(-1e19)), + ] { + let mut args = json!({ "path": "/a.md" }); + args[name] = passed.clone(); + let (kind, message) = refused(&args); + assert_eq!(kind, ToolErrorKind::InvalidArguments); + assert_eq!( + message, + format!( + "std_Read: `{name}` must be a JSON integer that fits in 64 bits, \ + and the call passed {passed}" + ) + ); + } +} + +#[test] +fn a_float_limit_below_one_is_refused_with_the_value_as_given() { + let (kind, message) = refused(&json!({ "path": "/a.md", "limit": 0.0 })); + assert_eq!(kind, ToolErrorKind::InvalidArguments); + assert_eq!( + message, + "std_Read: `limit` must be at least 1, and the call passed 0.0" + ); +} + +#[test] +fn an_undeclared_key_is_refused_naming_it_and_the_parameters() { + let (kind, message) = refused(&json!({ "path": "/a.md", "lines": 2 })); + assert_eq!(kind, ToolErrorKind::InvalidArguments); + assert_eq!( + message, + "std_Read: `lines` is not a parameter of this tool. \ + The parameters are `limit`, `offset` and `path`." + ); +} + +#[test] +fn a_blank_path_is_refused_before_any_io() { + for (blank, shown) in [("", ""), (" ", " ")] { + let (kind, message) = refused(&json!({ "path": blank })); + assert_eq!(kind, ToolErrorKind::InvalidArguments); + assert_eq!( + message, + format!( + "std_Read: `path` is empty or only whitespace, and the call passed \"{shown}\"" + ) + ); + } +} + +#[test] +fn a_missing_path_is_refused_by_the_shared_check() { + let (kind, message) = refused(&json!({})); + assert_eq!(kind, ToolErrorKind::InvalidArguments); + assert_eq!( + message, + "std_Read: `path` is required, and the call left it out" + ); +} + +#[test] +fn the_unknown_key_is_refused_before_the_blank_path() { + let (_, message) = refused(&json!({ "path": " ", "bogus": 1 })); + assert!(message.contains("`bogus` is not a parameter"), "{message}"); +} + +#[test] +fn the_blank_path_is_refused_before_the_limit() { + let (_, message) = refused(&json!({ "path": "", "limit": 0 })); + assert!( + message.contains("`path` is empty or only whitespace"), + "{message}" + ); +} + +#[test] +fn the_limit_is_refused_before_the_file_is_looked_up() { + let (kind, message) = refused(&json!({ "path": "/missing.md", "limit": 0 })); + assert_eq!(kind, ToolErrorKind::InvalidArguments); + assert!(message.contains("`limit` must be at least 1"), "{message}"); +} + +#[test] +fn a_missing_file_is_a_plain_message_and_not_an_invalid_arguments_error() { + let (kind, message) = refused(&json!({ "path": "/missing.md" })); + assert_ne!(kind, ToolErrorKind::InvalidArguments); + assert_eq!(message, "File not found: /missing.md"); +} + +#[test] +fn a_null_offset_and_a_null_limit_read_as_absent() { + let call = seeded("/a.md", b"one\ntwo\n"); + let plain = text_of(read(&call, &json!({ "path": "/a.md" }))); + let nulls = text_of(read( + &call, + &json!({ "path": "/a.md", "offset": null, "limit": null }), + )); + assert_eq!(plain, nulls); + assert_eq!(plain, " 1|one\n 2|two\n"); +} + +#[test] +fn a_null_argument_list_is_refused_for_its_missing_path() { + let (kind, message) = refused(&Value::Null); + assert_eq!(kind, ToolErrorKind::InvalidArguments); + assert_eq!( + message, + "std_Read: `path` is required, and the call left it out" + ); +} diff --git a/crates/plugin-std/src/read/tests/failures.rs b/crates/plugin-std/src/read/tests/failures.rs new file mode 100644 index 000000000..c3e9d56c7 --- /dev/null +++ b/crates/plugin-std/src/read/tests/failures.rs @@ -0,0 +1,218 @@ +//! What Read says when it cannot read, and how it treats the files it can: +//! encodings, binary data, empty files, and the size ceiling. + +use runvil_plugin::{FileType, OutputTrust}; +use serde_json::json; + +use super::{failure_of, new_call, read, read_capped, seed, seeded, text_of}; +use crate::read::{BYTE_CAP, check_type}; + +#[test] +fn a_missing_path_fails_with_the_not_found_text() { + let call = new_call(); + assert_eq!( + failure_of(read(&call, &json!({ "path": "/nope.md" }))), + "File not found: /nope.md" + ); +} + +#[test] +fn a_relative_path_is_echoed_as_written() { + let call = new_call(); + assert_eq!( + failure_of(read(&call, &json!({ "path": "notes/a.md" }))), + "File not found: notes/a.md" + ); +} + +#[test] +fn a_directory_fails_with_the_directory_text() { + let call = seeded("/dir/a.md", b"x\n"); + assert_eq!( + failure_of(read(&call, &json!({ "path": "/dir" }))), + "Path is a directory, not a file: /dir" + ); +} + +#[test] +fn the_file_type_decision_covers_every_kind_of_node() { + assert_eq!(check_type("/p", FileType::File), Ok(())); + assert_eq!(check_type("/p", FileType::Symlink), Ok(())); + assert_eq!( + check_type("/p", FileType::Directory), + Err("Path is a directory, not a file: /p".to_owned()) + ); + for special in [ + FileType::Fifo, + FileType::Socket, + FileType::CharDevice, + FileType::BlockDevice, + ] { + assert_eq!( + check_type("/p", special), + Err("Path is neither a file nor a directory: /p".to_owned()), + "{special:?}" + ); + } +} + +#[test] +fn a_binary_file_is_refused_by_its_content_not_its_name() { + let call = seeded("/data.md", &[0x00, 0x01, 0x02, 0x03, 0xFF]); + assert_eq!( + failure_of(read(&call, &json!({ "path": "/data.md" }))), + "File content is binary data, not text: /data.md" + ); + let text = seeded("/data.bin", b"plain text\n"); + assert_eq!( + text_of(read(&text, &json!({ "path": "/data.bin" }))), + " 1|plain text\n" + ); +} + +#[test] +fn a_file_one_byte_over_the_ceiling_is_refused_and_never_truncated() { + let call = seeded("/big.txt", &vec![b'a'; 16_777_217]); + assert_eq!( + failure_of(read(&call, &json!({ "path": "/big.txt" }))), + "File is too large to read: /big.txt is over the limit of 16777216 bytes." + ); +} + +#[test] +fn a_file_exactly_at_the_ceiling_is_read() { + let call = seeded("/edge.txt", &vec![b'a'; 16_777_216]); + let text = text_of(read(&call, &json!({ "path": "/edge.txt" }))); + assert!(text.starts_with(" 1|aaaa"), "the line is numbered"); + assert!( + text.contains("[Line 1 is 16777216 bytes and is cut after 99992 bytes."), + "{}", + &text[text.len() - 150..] + ); +} + +#[test] +fn the_ceiling_is_checked_before_the_binary_sniff() { + let call = seeded("/big.bin", &vec![0u8; 16_777_217]); + assert!( + failure_of(read(&call, &json!({ "path": "/big.bin" }))).starts_with("File is too large") + ); +} + +#[test] +fn an_empty_file_answers_file_is_empty_whatever_the_window() { + let call = seeded("/empty.md", b""); + for args in [ + json!({ "path": "/empty.md" }), + json!({ "path": "/empty.md", "offset": 500 }), + json!({ "path": "/empty.md", "offset": -3, "limit": 2 }), + json!({ "path": "/empty.md", "limit": 1 }), + ] { + assert_eq!(text_of(read(&call, &args)), "File is empty.", "{args}"); + } +} + +#[test] +fn a_file_of_only_a_byte_order_mark_is_empty() { + let call = seeded("/mark.md", b"\xEF\xBB\xBF"); + assert_eq!( + text_of(read(&call, &json!({ "path": "/mark.md", "offset": 9 }))), + "File is empty." + ); + let utf16 = seeded("/mark16.md", &[0xFF, 0xFE]); + assert_eq!( + text_of(read(&utf16, &json!({ "path": "/mark16.md" }))), + "File is empty." + ); +} + +#[test] +fn a_utf8_byte_order_mark_is_not_part_of_the_first_line() { + let call = seeded("/mark.md", b"\xEF\xBB\xBFhello\n"); + assert_eq!( + text_of(read(&call, &json!({ "path": "/mark.md" }))), + " 1|hello\n" + ); +} + +#[test] +fn a_utf16_file_is_decoded_and_the_note_names_its_byte_order() { + let mut le = vec![0xFF, 0xFE]; + let mut be = vec![0xFE, 0xFF]; + for unit in "a\nb\n".encode_utf16() { + le.extend_from_slice(&unit.to_le_bytes()); + be.extend_from_slice(&unit.to_be_bytes()); + } + for (name, bytes, order) in [("/le.txt", le, "UTF-16LE"), ("/be.txt", be, "UTF-16BE")] { + let call = seeded(name, &bytes); + assert_eq!( + text_of(read(&call, &json!({ "path": name }))), + format!(" 1|a\n 2|b\n[Note: the file is {order} and was decoded to text.]"), + "{name}" + ); + } +} + +#[test] +fn a_utf16_file_without_a_mark_is_binary() { + let call = seeded("/bare16.txt", &[b'h', 0, b'i', 0, b'\n', 0]); + assert_eq!( + failure_of(read(&call, &json!({ "path": "/bare16.txt" }))), + "File content is binary data, not text: /bare16.txt" + ); +} + +#[test] +fn an_invalid_byte_is_shown_as_the_replacement_character_with_a_note() { + let call = seeded("/latin.txt", &[0x61, 0xFF, 0x62, 0x0A]); + assert_eq!( + text_of(read(&call, &json!({ "path": "/latin.txt" }))), + " 1|a\u{FFFD}b\n\ + [Note: 1 invalid UTF-8 sequence in this file were shown as U+FFFD.]" + ); +} + +#[test] +fn a_file_that_ends_mid_sequence_counts_one_replacement() { + let call = seeded("/cut.txt", &[b'a', b'\n', b'b', 0xE2, 0x82]); + assert_eq!( + text_of(read(&call, &json!({ "path": "/cut.txt" }))), + " 1|a\n 2|b\u{FFFD}\n\ + [Note: 1 invalid UTF-8 sequence in this file were shown as U+FFFD.]" + ); +} + +#[test] +fn the_replacement_note_comes_before_the_trailer() { + let call = seeded("/two.txt", &[0x61, 0xFF, 0x0A, 0x62, 0x0A]); + assert_eq!( + text_of(read(&call, &json!({ "path": "/two.txt", "limit": 1 }))), + " 1|a\u{FFFD}\n\ + [Note: 1 invalid UTF-8 sequence in this file were shown as U+FFFD.]\n\ + [Showing lines 1-1 of 2. Use offset=2 to continue.]" + ); +} + +#[test] +fn every_successful_read_is_untrusted_output() { + let call = seeded("/a.md", b"one\ntwo\n"); + seed(&call, "/empty.md", b""); + seed(&call, "/long.md", "x".repeat(150_000).as_bytes()); + for args in [ + json!({ "path": "/a.md" }), + json!({ "path": "/a.md", "offset": 2 }), + json!({ "path": "/a.md", "limit": 1 }), + json!({ "path": "/empty.md" }), + json!({ "path": "/long.md" }), + ] { + let output = read(&call, &args).expect("the read succeeds"); + assert_eq!(output.trust(), OutputTrust::Untrusted, "{args}"); + } + let capped = read_capped(&call, "/a.md", None, None, 10).expect("a cut read succeeds"); + assert_eq!(capped.trust(), OutputTrust::Untrusted); +} + +#[test] +fn the_byte_cap_is_one_hundred_thousand_bytes() { + assert_eq!(BYTE_CAP, 100_000); +} diff --git a/crates/plugin-std/src/read/tests/paging.rs b/crates/plugin-std/src/read/tests/paging.rs new file mode 100644 index 000000000..ce6d9c0dd --- /dev/null +++ b/crates/plugin-std/src/read/tests/paging.rs @@ -0,0 +1,95 @@ +//! Paging: following each `Use offset=N` trailer until the read ends must +//! rebuild the file exactly, whatever the cap and the limit. + +use super::{numbered_file, read_capped, seeded, text_of}; +use crate::read::BYTE_CAP; + +/// Reads `/file.txt` page by page under `cap` and `limit`, following each +/// trailer's offset, and returns the text of every line it printed. +fn rebuild(file: &str, total: usize, cap: usize, limit: Option) -> Vec { + let call = seeded("/file.txt", file.as_bytes()); + let mut rebuilt: Vec = Vec::new(); + let mut offset: Option = None; + let mut pages = 0usize; + loop { + pages += 1; + assert!(pages <= total + 1, "paging does not advance"); + let text = text_of(read_capped(&call, "/file.txt", offset, limit, cap)); + let mut shown: Vec<&str> = text.split('\n').collect(); + if shown.last() == Some(&"") { + shown.pop(); + } + let trailer = shown + .last() + .copied() + .filter(|line| line.starts_with("[Showing lines ")); + if trailer.is_some() { + shown.pop(); + } + assert!(!shown.is_empty(), "a page is never empty: {text}"); + for line in shown { + let number = rebuilt.len() + 1; + let (prefix, content) = line.split_at(7); + assert_eq!(prefix, format!("{number:>6}|"), "page {pages}"); + rebuilt.push(content.to_owned()); + } + let Some(trailer) = trailer else { + break; + }; + let next: i64 = trailer + .split("offset=") + .nth(1) + .and_then(|rest| rest.split(' ').next()) + .and_then(|digits| digits.parse().ok()) + .unwrap_or_else(|| panic!("the trailer names an offset: {trailer}")); + assert_eq!( + usize::try_from(next).expect("the offset is positive"), + rebuilt.len() + 1, + "the trailer continues where the page stopped" + ); + offset = Some(next); + } + assert_eq!(rebuilt.len(), total, "the pages cover the file"); + rebuilt +} + +#[test] +fn following_the_trailers_rebuilds_the_file_under_every_cap_and_limit() { + for total in [1, 2, 5, 37, 2_000, 2_001] { + for final_newline in [true, false] { + let file = numbered_file(total, final_newline); + let expected: Vec = (1..=total).map(|n| format!("line {n}")).collect(); + for cap in [BYTE_CAP, 300, 90] { + for limit in [None, Some(50), Some(7)] { + let rebuilt = rebuild(&file, total, cap, limit); + assert_eq!( + rebuilt, expected, + "{total} lines, final newline {final_newline}, cap {cap}, limit {limit:?}" + ); + } + } + } + } +} + +#[test] +fn a_small_cap_makes_more_pages_than_a_large_one() { + let file = numbered_file(37, true); + let pages = |cap: usize| { + let call = seeded("/file.txt", file.as_bytes()); + let mut count = 0; + let mut offset: Option = None; + loop { + count += 1; + let text = text_of(read_capped(&call, "/file.txt", offset, None, cap)); + let Some(rest) = text.rsplit_once("offset=") else { + return count; + }; + let digits = rest.1.trim_end_matches(" to continue.]"); + offset = Some(digits.parse().expect("the offset is a number")); + } + }; + assert_eq!(pages(BYTE_CAP), 1); + assert!(pages(90) > pages(300)); + assert!(pages(300) > 1); +} diff --git a/crates/plugin-std/src/read/tests/render.rs b/crates/plugin-std/src/read/tests/render.rs new file mode 100644 index 000000000..f2d820645 --- /dev/null +++ b/crates/plugin-std/src/read/tests/render.rs @@ -0,0 +1,304 @@ +//! The render: the numbered lines, the byte cap, the trailer, the cut of a +//! first line that alone passes the cap, and the notes. + +use super::{read_capped, seeded, text_of}; +use crate::read::BYTE_CAP; +use crate::read::render::{notes, render}; +use crate::read::window::Window; + +/// The three lines most render tests print. Each costs 11 bytes: a +/// seven-byte prefix, three bytes of text, and a newline. +const LINES: [&str; 3] = ["aaa", "bbb", "ccc"]; + +/// The window over every line of [`LINES`]. +const ALL: Window = Window { first: 1, last: 3 }; + +#[test] +fn every_line_prints_as_a_six_wide_number_a_pipe_and_the_text() { + let text = render(&LINES, &ALL, &[], BYTE_CAP); + assert_eq!(text, " 1|aaa\n 2|bbb\n 3|ccc\n"); +} + +#[test] +fn a_window_in_the_middle_keeps_the_real_line_numbers() { + let text = render(&LINES, &Window { first: 2, last: 3 }, &[], BYTE_CAP); + assert_eq!(text, " 2|bbb\n 3|ccc\n"); +} + +#[test] +fn a_window_that_stops_early_ends_with_the_limit_trailer() { + let text = render(&LINES, &Window { first: 2, last: 2 }, &[], BYTE_CAP); + assert_eq!( + text, + " 2|bbb\n[Showing lines 2-2 of 3. Use offset=3 to continue.]" + ); +} + +#[test] +fn lines_that_exactly_fill_the_cap_all_print_and_no_trailer_follows() { + let text = render(&LINES, &ALL, &[], 33); + assert_eq!(text, " 1|aaa\n 2|bbb\n 3|ccc\n"); +} + +#[test] +fn the_line_that_would_pass_the_cap_is_left_out_and_the_trailer_names_the_cap() { + let text = render(&LINES, &ALL, &[], 32); + assert_eq!( + text, + " 1|aaa\n 2|bbb\n\ + [Showing lines 1-2 of 3 (32-byte limit). Use offset=3 to continue.]" + ); +} + +#[test] +fn the_cap_counts_each_prefix_the_text_and_the_newline() { + for (cap, printed) in [(11, 1), (21, 1), (22, 2), (32, 2), (33, 3)] { + let text = render(&LINES, &ALL, &[], cap); + let shown = text.lines().filter(|line| line.contains('|')).count(); + assert_eq!(shown, printed, "cap {cap}: {text}"); + } +} + +#[test] +fn the_cap_trailer_wins_over_the_limit_when_the_cap_stops_first() { + let text = render(&LINES, &Window { first: 1, last: 2 }, &[], 11); + assert_eq!( + text, + " 1|aaa\n[Showing lines 1-1 of 3 (11-byte limit). Use offset=2 to continue.]" + ); +} + +#[test] +fn a_first_line_that_alone_passes_the_cap_is_cut_and_the_trailer_says_so() { + let text = render(&LINES, &ALL, &[], 10); + assert_eq!( + text, + " 1|aa\n[Line 1 is 3 bytes and is cut after 2 bytes. \ + No parameter reads the rest of a line. Use offset=2 to continue.]" + ); +} + +#[test] +fn a_cut_first_line_on_the_last_line_of_the_file_offers_no_next_offset() { + let text = render(&LINES, &Window { first: 3, last: 3 }, &[], 10); + assert_eq!( + text, + " 3|cc\n[Line 3 is 3 bytes and is cut after 2 bytes. \ + No parameter reads the rest of a line.]" + ); +} + +#[test] +fn a_cap_below_the_prefix_keeps_no_text_and_does_not_panic() { + let text = render(&LINES, &ALL, &[], 7); + assert_eq!( + text, + " 1|\n[Line 1 is 3 bytes and is cut after 0 bytes. \ + No parameter reads the rest of a line. Use offset=2 to continue.]" + ); +} + +#[test] +fn notes_come_after_the_lines_and_before_the_trailer_each_on_its_own_line() { + let note_lines = ["[Note: one]".to_owned(), "[Note: two]".to_owned()]; + let with_trailer = render(&LINES, &Window { first: 1, last: 1 }, ¬e_lines, BYTE_CAP); + assert_eq!( + with_trailer, + " 1|aaa\n[Note: one]\n[Note: two]\n\ + [Showing lines 1-1 of 3. Use offset=2 to continue.]" + ); + let without = render(&LINES, &ALL, ¬e_lines, BYTE_CAP); + assert_eq!( + without, + " 1|aaa\n 2|bbb\n 3|ccc\n[Note: one]\n[Note: two]" + ); +} + +#[test] +fn a_utf16_file_adds_a_note_naming_its_byte_order() { + assert_eq!( + notes(Some("UTF-16LE"), 0), + ["[Note: the file is UTF-16LE and was decoded to text.]"] + ); + assert_eq!( + notes(Some("UTF-16BE"), 0), + ["[Note: the file is UTF-16BE and was decoded to text.]"] + ); +} + +#[test] +fn replaced_bytes_add_a_note_that_counts_the_sequences() { + assert_eq!( + notes(None, 1), + ["[Note: 1 invalid UTF-8 sequence in this file were shown as U+FFFD.]"] + ); + assert_eq!( + notes(None, 3), + ["[Note: 3 invalid UTF-8 sequences in this file were shown as U+FFFD.]"] + ); +} + +#[test] +fn the_utf16_note_comes_before_the_replacement_note_and_clean_text_has_none() { + assert_eq!( + notes(Some("UTF-16LE"), 2), + [ + "[Note: the file is UTF-16LE and was decoded to text.]", + "[Note: 2 invalid UTF-8 sequences in this file were shown as U+FFFD.]" + ] + ); + assert!(notes(None, 0).is_empty()); +} + +#[test] +fn five_thousand_short_lines_stop_at_2083_lines_under_the_default_cap() { + let file = format!("{}\n", "x".repeat(40)).repeat(5_000); + let call = seeded("/many.txt", file.as_bytes()); + let text = text_of(read_capped(&call, "/many.txt", None, None, BYTE_CAP)); + let trailer = + "[Showing lines 1-2083 of 5000 (100000-byte limit). Use offset=2084 to continue.]"; + assert!( + text.ends_with(&format!("\n{trailer}")), + "the trailer ends the read" + ); + let numbered = text + .lines() + .filter(|line| line.as_bytes().get(6) == Some(&b'|')); + assert_eq!(numbered.count(), 2_083); + assert_eq!( + text.len() - trailer.len(), + 2_083 * 48, + "each numbered line is 7 + 40 + 1 bytes and the trailer is not counted" + ); + assert!(text.len() <= 100_120, "{}", text.len()); +} + +/// What Read answers for a file whose first line is `line`, followed by +/// a second line when `later` is set, split into the line it printed +/// after its prefix and the trailer. +fn first_line_cut(line: &str, later: bool) -> (String, String) { + let mut file = line.to_owned(); + file.push('\n'); + if later { + file.push_str("next\n"); + } + let call = seeded("/long.txt", file.as_bytes()); + let text = text_of(read_capped(&call, "/long.txt", None, None, BYTE_CAP)); + let (shown, trailer) = text.split_once('\n').expect("a cut read has a trailer"); + let kept = shown + .strip_prefix(" 1|") + .expect("the first line is numbered"); + (kept.to_owned(), trailer.to_owned()) +} + +#[test] +fn a_150000_byte_ascii_line_keeps_99992_bytes() { + let line = "x".repeat(150_000); + let (kept, trailer) = first_line_cut(&line, false); + assert_eq!(kept.len(), 99_992); + assert!(line.starts_with(&kept)); + assert_eq!( + trailer, + "[Line 1 is 150000 bytes and is cut after 99992 bytes. \ + No parameter reads the rest of a line.]" + ); + let (_, trailer) = first_line_cut(&line, true); + assert_eq!( + trailer, + "[Line 1 is 150000 bytes and is cut after 99992 bytes. \ + No parameter reads the rest of a line. Use offset=2 to continue.]" + ); +} + +#[test] +fn the_cut_numbered_line_fills_the_cap_exactly_for_ascii() { + let call = seeded("/long.txt", "x".repeat(150_000).as_bytes()); + let text = text_of(read_capped(&call, "/long.txt", None, None, BYTE_CAP)); + let first = text.split_once('\n').expect("one numbered line").0; + assert_eq!(first.len() + 1, 100_000); +} + +#[test] +fn fifty_thousand_cjk_characters_keep_99990_bytes() { + let line = "\u{4E2D}".repeat(50_000); + let (kept, trailer) = first_line_cut(&line, false); + assert_eq!(kept.len(), 99_990); + assert_eq!(kept.chars().count(), 33_330); + assert!( + trailer.contains("is 150000 bytes and is cut after 99990 bytes."), + "{trailer}" + ); +} + +#[test] +fn a_line_of_two_byte_characters_keeps_99992_bytes() { + let line = "\u{E9}".repeat(75_000); + let (kept, trailer) = first_line_cut(&line, false); + assert_eq!(kept.len(), 99_992); + assert!(line.starts_with(&kept)); + assert!(trailer.contains("cut after 99992 bytes."), "{trailer}"); +} + +#[test] +fn a_line_of_four_byte_emoji_keeps_24998_of_them() { + let line = "\u{1F600}".repeat(37_500); + let (kept, trailer) = first_line_cut(&line, false); + assert_eq!(kept.len(), 99_992); + assert_eq!(kept.chars().count(), 24_998); + assert!(trailer.contains("cut after 99992 bytes."), "{trailer}"); +} + +#[test] +fn a_cut_at_a_later_line_names_that_line() { + let file = format!("short\n{}\ntail\n", "x".repeat(150_000)); + let call = seeded("/long.txt", file.as_bytes()); + let text = text_of(read_capped(&call, "/long.txt", Some(2), None, BYTE_CAP)); + let trailer = text.split_once('\n').expect("a cut read has a trailer").1; + assert_eq!( + trailer, + "[Line 2 is 150000 bytes and is cut after 99992 bytes. \ + No parameter reads the rest of a line. Use offset=3 to continue.]" + ); +} + +#[test] +fn a_line_that_only_just_fits_is_not_cut() { + let line = "x".repeat(BYTE_CAP - 8); + let call = seeded("/fit.txt", format!("{line}\n").as_bytes()); + let text = text_of(read_capped(&call, "/fit.txt", None, None, BYTE_CAP)); + assert_eq!(text, format!(" 1|{line}\n")); +} + +#[test] +fn a_cap_sweep_never_splits_a_character_and_the_notice_counts_the_kept_bytes() { + let line = "a\u{E9}\u{20AC}\u{1F600}".repeat(30); + let call = seeded("/mixed.txt", format!("{line}\n").as_bytes()); + for cap in 12..=60 { + let text = text_of(read_capped(&call, "/mixed.txt", None, None, cap)); + let (shown, trailer) = text.split_once('\n').expect("a cut read has a trailer"); + let kept = shown + .strip_prefix(" 1|") + .expect("the first line is numbered"); + let allowance = cap - 8; + assert!(line.starts_with(kept), "cap {cap}: a prefix of the line"); + assert!( + kept.len() <= allowance, + "cap {cap}: {} bytes kept", + kept.len() + ); + assert!( + allowance - kept.len() < 4, + "cap {cap}: the cut stops within one character of the allowance" + ); + assert_eq!( + trailer, + format!( + "[Line 1 is {} bytes and is cut after {} bytes. \ + No parameter reads the rest of a line.]", + line.len(), + kept.len() + ), + "cap {cap}" + ); + } +} diff --git a/crates/plugin-std/src/read/tests/window.rs b/crates/plugin-std/src/read/tests/window.rs new file mode 100644 index 000000000..90deff5ad --- /dev/null +++ b/crates/plugin-std/src/read/tests/window.rs @@ -0,0 +1,177 @@ +//! The window: which lines an offset and a limit pick, and what Read +//! prints for them. + +use super::{failure_of, numbered_file, read, read_capped, seeded, text_of}; +use crate::read::BYTE_CAP; +use crate::read::window::{Window, window}; +use serde_json::json; + +/// The window of a 10-line file, as first and last line numbers. +fn ten(offset: Option, limit: Option) -> Option<(usize, usize)> { + window(10, offset, limit).map(|picked| (picked.first, picked.last)) +} + +/// An offset, a limit, and the first and last line they pick. +type Case = (Option, Option, Option<(usize, usize)>); + +#[test] +fn the_window_table_on_a_ten_line_file() { + let table: [Case; 13] = [ + (None, None, Some((1, 10))), + (Some(0), None, Some((1, 10))), + (Some(1), None, Some((1, 10))), + (Some(10), None, Some((10, 10))), + (Some(-1), None, Some((10, 10))), + (Some(-2), None, Some((9, 10))), + (Some(-100), None, Some((1, 10))), + (Some(-2), Some(5), Some((9, 10))), + (None, Some(3), Some((1, 3))), + (Some(4), Some(3), Some((4, 6))), + (Some(9), Some(5), Some((9, 10))), + (Some(11), None, None), + (Some(11), Some(3), None), + ]; + for (offset, limit, expected) in table { + assert_eq!( + ten(offset, limit), + expected, + "offset {offset:?}, limit {limit:?}" + ); + } +} + +#[test] +fn the_extremes_of_both_arguments_do_not_overflow() { + assert_eq!(ten(Some(i64::MIN), None), Some((1, 10))); + assert_eq!(ten(Some(i64::MAX), None), None); + assert_eq!(ten(None, Some(usize::MAX)), Some((1, 10))); + assert_eq!(ten(Some(5), Some(usize::MAX)), Some((5, 10))); + assert_eq!(ten(Some(-3), Some(usize::MAX)), Some((8, 10))); +} + +#[test] +fn a_one_line_file_has_one_window() { + assert_eq!(window(1, None, None), Some(Window { first: 1, last: 1 })); + assert_eq!( + window(1, Some(-1), None), + Some(Window { first: 1, last: 1 }) + ); + assert_eq!(window(1, Some(2), None), None); +} + +#[test] +fn a_whole_file_read_numbers_every_line_and_prints_no_trailer() { + let call = seeded("/notes.md", numbered_file(3, true).as_bytes()); + let text = text_of(read(&call, &json!({ "path": "/notes.md" }))); + assert_eq!(text, " 1|line 1\n 2|line 2\n 3|line 3\n"); +} + +#[test] +fn a_file_without_a_final_newline_has_the_same_lines() { + let with = seeded("/a.md", numbered_file(3, true).as_bytes()); + let without = seeded("/a.md", numbered_file(3, false).as_bytes()); + let args = json!({ "path": "/a.md" }); + assert_eq!(text_of(read(&with, &args)), text_of(read(&without, &args))); +} + +#[test] +fn a_blank_last_line_counts_as_a_line() { + let call = seeded("/a.md", b"a\n\n"); + let text = text_of(read(&call, &json!({ "path": "/a.md" }))); + assert_eq!(text, " 1|a\n 2|\n"); +} + +#[test] +fn carriage_returns_are_not_part_of_a_line() { + let call = seeded("/a.md", b"a\r\nb\r\n"); + let text = text_of(read(&call, &json!({ "path": "/a.md" }))); + assert_eq!(text, " 1|a\n 2|b\n"); +} + +#[test] +fn a_lone_carriage_return_inside_a_line_stays_in_the_line() { + let call = seeded("/a.md", b"first\rNEEDLE\rthird\n"); + let text = text_of(read(&call, &json!({ "path": "/a.md" }))); + assert_eq!(text, " 1|first\rNEEDLE\rthird\n"); +} + +#[test] +fn a_limit_that_stops_before_the_end_names_the_next_offset() { + let call = seeded("/ten.md", numbered_file(10, true).as_bytes()); + let text = text_of(read( + &call, + &json!({ "path": "/ten.md", "offset": 4, "limit": 3 }), + )); + assert_eq!( + text, + " 4|line 4\n 5|line 5\n 6|line 6\n\ + [Showing lines 4-6 of 10. Use offset=7 to continue.]" + ); +} + +#[test] +fn a_read_that_reaches_the_last_line_prints_no_trailer() { + let call = seeded("/ten.md", numbered_file(10, true).as_bytes()); + let tail = " 9|line 9\n 10|line 10\n"; + for args in [ + json!({ "path": "/ten.md", "offset": 9, "limit": 5 }), + json!({ "path": "/ten.md", "offset": 9 }), + json!({ "path": "/ten.md", "offset": -2 }), + json!({ "path": "/ten.md", "offset": -2, "limit": 5 }), + json!({ "path": "/ten.md", "offset": 9, "limit": 2 }), + ] { + assert_eq!(text_of(read(&call, &args)), tail, "{args}"); + } +} + +#[test] +fn a_limit_that_ends_exactly_on_the_last_line_prints_no_trailer() { + let call = seeded("/ten.md", numbered_file(10, true).as_bytes()); + let text = text_of(read(&call, &json!({ "path": "/ten.md", "limit": 10 }))); + assert!(!text.contains("[Showing"), "{text}"); +} + +#[test] +fn the_negative_offsets_read_the_tail_of_the_file() { + let call = seeded("/ten.md", numbered_file(10, true).as_bytes()); + let last = text_of(read(&call, &json!({ "path": "/ten.md", "offset": -1 }))); + assert_eq!(last, " 10|line 10\n"); + let all = text_of(read(&call, &json!({ "path": "/ten.md", "offset": -100 }))); + assert_eq!(all, text_of(read(&call, &json!({ "path": "/ten.md" })))); +} + +#[test] +fn an_offset_of_zero_reads_as_the_first_line() { + let call = seeded("/ten.md", numbered_file(10, true).as_bytes()); + let zero = text_of(read( + &call, + &json!({ "path": "/ten.md", "offset": 0, "limit": 2 }), + )); + assert_eq!( + zero, + " 1|line 1\n 2|line 2\n[Showing lines 1-2 of 10. Use offset=3 to continue.]" + ); +} + +#[test] +fn an_offset_past_the_end_fails_with_the_line_count() { + let ten = seeded("/notes.md", numbered_file(10, true).as_bytes()); + assert_eq!( + failure_of(read(&ten, &json!({ "path": "/notes.md", "offset": 11 }))), + "Offset 11 is beyond the end of /notes.md, which has 10 lines." + ); + let one = seeded("/one.md", b"only\n"); + assert_eq!( + failure_of(read(&one, &json!({ "path": "/one.md", "offset": 2 }))), + "Offset 2 is beyond the end of /one.md, which has 1 line." + ); +} + +#[test] +fn the_line_number_widens_past_six_digits_and_the_prefix_stays_in_the_cap() { + let call = seeded("/many.txt", "\n".repeat(1_000_000).as_bytes()); + let last = text_of(read_capped(&call, "/many.txt", Some(-1), None, BYTE_CAP)); + assert_eq!(last, "1000000|\n"); + let two = text_of(read_capped(&call, "/many.txt", Some(-2), None, BYTE_CAP)); + assert_eq!(two, "999999|\n1000000|\n"); +} diff --git a/crates/plugin-std/src/read/window.rs b/crates/plugin-std/src/read/window.rs new file mode 100644 index 000000000..f20c270be --- /dev/null +++ b/crates/plugin-std/src/read/window.rs @@ -0,0 +1,41 @@ +//! The window: which lines of a file an offset and a limit pick. + +/// A run of lines, as the numbers of its first and last line, counting +/// from 1. Both ends are inside the file. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) struct Window { + /// The first line, at least 1. + pub(super) first: usize, + /// The last line, at least `first` and at most the line count. + pub(super) last: usize, +} + +/// The window an `offset` and a `limit` pick over a file of `total` lines. +/// +/// `total` is at least 1, and `limit`, when there is one, is at least 1. +/// An absent offset starts at line 1, and so does an offset of `0`. A +/// positive offset starts there, and picks nothing (`None`) when it is +/// past the last line. A negative offset counts back from the end, so +/// `-1` is the last line, and it never picks nothing: it stops at line 1 +/// when it reaches past the start. The window ends at the last line, or +/// `limit` lines after its start when that is sooner. +pub(super) fn window(total: usize, offset: Option, limit: Option) -> Option { + let first = match offset { + None | Some(0) => 1, + Some(offset) if offset > 0 => { + let first = usize::try_from(offset).unwrap_or(usize::MAX); + if first > total { + return None; + } + first + } + Some(offset) => { + let back = usize::try_from(offset.unsigned_abs()).unwrap_or(usize::MAX); + total.saturating_sub(back) + 1 + } + }; + let last = limit.map_or(total, |limit| { + first.saturating_add(limit.saturating_sub(1)).min(total) + }); + Some(Window { first, last }) +} diff --git a/crates/plugin-std/src/refusal.rs b/crates/plugin-std/src/refusal.rs new file mode 100644 index 000000000..f6c0ab5a6 --- /dev/null +++ b/crates/plugin-std/src/refusal.rs @@ -0,0 +1,60 @@ +//! The sentence a model reads when a file operation fails. +//! +//! Every file tool turns a [`VfsError`] into one of these, so the same +//! failure reads the same in each tool. The path in a sentence is always +//! the path as the model wrote it, never the canonical path the error +//! holds, so a relative `notes/a.md` is not echoed back as `/notes/a.md`. + +use runvil_plugin::VfsError; + +/// The model-readable sentence for `error`, from the tool called `tool` +/// (the name the model uses) on `path` (as the model wrote it). +/// +/// A sentence names no tool other than `tool`. +pub(crate) fn refusal(tool: &str, path: &str, error: &VfsError) -> String { + match error { + VfsError::NotFound { .. } => format!("File not found: {path}"), + VfsError::IsADirectory { .. } => format!("Path is a directory, not a file: {path}"), + VfsError::NotADirectory { .. } => not_a_directory(path), + VfsError::Conflict { .. } => format!( + "{path} was changed, or is being changed, by another call in this round, so it \ + cannot be read in this round. Call {tool} again in your next message." + ), + VfsError::Io { kind, message, .. } => format!("I/O failure on {path} ({kind}): {message}"), + VfsError::PermissionDenied { reason, .. } | VfsError::ApprovalRequired { reason, .. } => { + reason.clone() + } + other => without_label(&other.to_string()), + } +} + +/// `text` without its leading label: the text after the first `": "`, or +/// the whole text when it has none. +fn without_label(text: &str) -> String { + text.split_once(": ") + .map_or(text, |(_, rest)| rest) + .to_owned() +} + +/// The sentence for a root the model named that does not exist. +/// +/// A tool that starts from a directory words its own missing root, because +/// the `NotFound` sentence above says a file was not found. +pub(crate) fn missing_root(path: &str) -> String { + format!("Path does not exist: {path}") +} + +/// The sentence for a path that names something other than a directory +/// where a directory was needed. +pub(crate) fn not_a_directory(path: &str) -> String { + format!("Path is not a directory: {path}") +} + +/// The sentence for a path that is neither a file nor a directory, such +/// as a pipe or a device. +pub(crate) fn neither_file_nor_directory(path: &str) -> String { + format!("Path is neither a file nor a directory: {path}") +} + +#[cfg(test)] +mod tests; diff --git a/crates/plugin-std/src/refusal/tests.rs b/crates/plugin-std/src/refusal/tests.rs new file mode 100644 index 000000000..17d6b5ef4 --- /dev/null +++ b/crates/plugin-std/src/refusal/tests.rs @@ -0,0 +1,338 @@ +//! Unit tests for the refusal texts: one sentence per `VfsError`, byte +//! for byte, with the path echoed as the model wrote it. + +use std::io::ErrorKind; + +use runvil_plugin::{PathReason, VfsError}; + +use super::{missing_root, neither_file_nor_directory, not_a_directory, refusal}; + +/// The path the model wrote, and the canonical path every error below +/// carries, which differs so a test can tell which one a sentence echoes. +const WRITTEN: &str = "notes/a.md"; +const CANONICAL: &str = "/canonical/a.md"; + +/// What `refusal` says for `error` when `Read` was called on +/// `notes/a.md`. +fn say(error: &VfsError) -> String { + refusal("Read", WRITTEN, error) +} + +#[test] +fn not_found_says_the_file_was_not_found_with_the_path_as_written() { + let error = VfsError::NotFound { + path: CANONICAL.to_owned(), + }; + assert_eq!(say(&error), "File not found: notes/a.md"); +} + +#[test] +fn a_directory_where_a_file_was_named_says_so() { + let error = VfsError::IsADirectory { + path: CANONICAL.to_owned(), + }; + assert_eq!(say(&error), "Path is a directory, not a file: notes/a.md"); +} + +#[test] +fn a_file_where_a_directory_was_named_says_so() { + let error = VfsError::NotADirectory { + path: CANONICAL.to_owned(), + }; + assert_eq!(say(&error), "Path is not a directory: notes/a.md"); +} + +#[test] +fn a_conflict_says_the_path_changed_this_round_and_names_the_tool_to_call_again() { + let error = VfsError::Conflict { + path: CANONICAL.to_owned(), + detail: "write by another call".to_owned(), + }; + assert_eq!( + say(&error), + "notes/a.md was changed, or is being changed, by another call in this round, so it \ + cannot be read in this round. Call Read again in your next message." + ); + assert_eq!( + refusal("Grep", "/src", &error), + "/src was changed, or is being changed, by another call in this round, so it \ + cannot be read in this round. Call Grep again in your next message." + ); +} + +#[test] +fn an_io_failure_names_the_path_the_kind_and_the_message() { + let error = VfsError::Io { + path: CANONICAL.to_owned(), + kind: ErrorKind::TimedOut, + raw_os_error: Some(110), + message: "the disk did not answer".to_owned(), + }; + assert_eq!( + say(&error), + "I/O failure on notes/a.md (timed out): the disk did not answer" + ); +} + +#[test] +fn a_denied_or_unapproved_read_passes_the_policys_reason_through_unchanged() { + let reason = "Mode::Plan allows only markdown paths, and /secret.rs is not one"; + let denied = VfsError::PermissionDenied { + path: CANONICAL.to_owned(), + reason: reason.to_owned(), + }; + assert_eq!(say(&denied), reason); + let approval = "Reading /vault needs approval (rule: vault)"; + let asked = VfsError::ApprovalRequired { + path: CANONICAL.to_owned(), + reason: approval.to_owned(), + }; + assert_eq!(say(&asked), approval); +} + +#[test] +fn every_other_variant_gives_its_display_text_without_the_leading_label() { + let cases: Vec<(VfsError, &str)> = vec![ + ( + VfsError::AlreadyExists { + path: CANONICAL.to_owned(), + }, + "/canonical/a.md", + ), + ( + VfsError::DirectoryNotEmpty { + path: CANONICAL.to_owned(), + }, + "/canonical/a.md", + ), + ( + VfsError::NotUtf8 { + path: CANONICAL.to_owned(), + valid_up_to: 4, + }, + "/canonical/a.md, valid up to byte 4", + ), + ( + VfsError::InvalidPath { + path: "a\\b".to_owned(), + reason: PathReason::Backslash, + }, + "path contains a backslash", + ), + ( + VfsError::InvalidRange { + path: CANONICAL.to_owned(), + reason: "the range is empty", + }, + "the range is empty", + ), + ( + VfsError::Anchor { + path: CANONICAL.to_owned(), + anchor: String::new(), + count: 0, + }, + "/canonical/a.md", + ), + ( + VfsError::Anchor { + path: CANONICAL.to_owned(), + anchor: "old".to_owned(), + count: 0, + }, + "anchor \"old\" was not found in /canonical/a.md, expected exactly one", + ), + ]; + for (error, expected) in cases { + assert_eq!(say(&error), expected, "{error:?}"); + } +} + +#[test] +fn the_state_and_backend_variants_also_drop_the_leading_label() { + let cases: Vec<(VfsError, &str)> = vec![ + ( + VfsError::ScopeEnded { + path: CANONICAL.to_owned(), + }, + "the run or tool call that owned this access has ended, so /canonical/a.md can no \ + longer be touched through it", + ), + ( + VfsError::ReadOnly { + path: CANONICAL.to_owned(), + }, + "/canonical/a.md is served read-only and cannot change", + ), + ( + VfsError::OutsideRoot { + path: CANONICAL.to_owned(), + detail: "/canonical/a.md is a link to /etc/passwd, which is outside the root" + .to_owned(), + }, + "/canonical/a.md is a link to /etc/passwd, which is outside the root", + ), + ( + VfsError::RootProtected { + path: "/".to_owned(), + }, + "the mounted root / cannot be removed, renamed, or replaced", + ), + (VfsError::NoStore, "the handle declares no store"), + ( + VfsError::CrossMount { + path: CANONICAL.to_owned(), + detail: "copy /a to /b".to_owned(), + }, + "copy /a to /b", + ), + ( + VfsError::Unsupported { + path: CANONICAL.to_owned(), + detail: "this backend cannot seek".to_owned(), + }, + "this backend cannot seek", + ), + ( + VfsError::Backend { + message: "the store is offline".to_owned(), + }, + "the store is offline", + ), + ]; + for (error, expected) in cases { + assert_eq!(say(&error), expected, "{error:?}"); + } +} + +#[test] +fn an_outside_root_error_reads_as_its_detail() { + let error = VfsError::OutsideRoot { + path: CANONICAL.to_owned(), + detail: "the link leaves the run's filesystem".to_owned(), + }; + assert_eq!(say(&error), "the link leaves the run's filesystem"); +} + +#[test] +fn a_display_text_with_no_label_is_kept_whole() { + let error = VfsError::Anchor { + path: CANONICAL.to_owned(), + anchor: "x".to_owned(), + count: 3, + }; + assert_eq!( + say(&error), + "anchor \"x\" occurs 3 times in /canonical/a.md, expected exactly one" + ); +} + +#[test] +fn a_relative_path_is_echoed_as_written_and_never_as_the_canonical_one() { + let error = VfsError::NotFound { + path: "/notes/a.md".to_owned(), + }; + let text = say(&error); + assert!(text.ends_with("notes/a.md"), "{text}"); + assert!(!text.contains("/notes/a.md"), "{text}"); +} + +#[test] +fn no_message_names_a_tool_other_than_the_one_called() { + let others = ["Write", "StrReplace", "Delete", "Glob", "Grep", "Shell"]; + let errors = [ + VfsError::NotFound { + path: CANONICAL.to_owned(), + }, + VfsError::IsADirectory { + path: CANONICAL.to_owned(), + }, + VfsError::NotADirectory { + path: CANONICAL.to_owned(), + }, + VfsError::Conflict { + path: CANONICAL.to_owned(), + detail: "x".to_owned(), + }, + VfsError::Io { + path: CANONICAL.to_owned(), + kind: ErrorKind::Other, + raw_os_error: None, + message: "x".to_owned(), + }, + VfsError::Anchor { + path: CANONICAL.to_owned(), + anchor: String::new(), + count: 0, + }, + VfsError::ReadOnly { + path: CANONICAL.to_owned(), + }, + VfsError::NoStore, + VfsError::DirectoryNotEmpty { + path: CANONICAL.to_owned(), + }, + ]; + for error in errors { + let text = say(&error); + for other in others { + assert!(!text.contains(other), "{error:?} says {text}"); + } + } + assert!(!neither_file_nor_directory("p").contains("Shell")); +} + +#[test] +fn a_root_that_does_not_exist_says_the_path_does_not_exist() { + assert_eq!( + missing_root("/home/dev/gone"), + "Path does not exist: /home/dev/gone" + ); + assert_eq!(missing_root("gone"), "Path does not exist: gone"); +} + +#[test] +fn a_missing_root_is_worded_apart_from_a_missing_file() { + // The `NotFound` sentence says file, which is wrong for a directory + // the model named, so the listing and search tools word the root + // themselves. + let error = VfsError::NotFound { + path: CANONICAL.to_owned(), + }; + assert_eq!(say(&error), "File not found: notes/a.md"); + assert_eq!(missing_root(WRITTEN), "Path does not exist: notes/a.md"); +} + +#[test] +fn the_not_a_directory_sentence_is_the_one_the_refusal_gives() { + let error = VfsError::NotADirectory { + path: CANONICAL.to_owned(), + }; + assert_eq!(say(&error), not_a_directory(WRITTEN)); + assert_eq!(not_a_directory("/a/b"), "Path is not a directory: /a/b"); +} + +#[test] +fn the_root_sentences_name_no_tool() { + for text in [missing_root("/p"), not_a_directory("/p")] { + for tool in [ + "Read", + "Write", + "StrReplace", + "Delete", + "Glob", + "Grep", + "Shell", + ] { + assert!(!text.contains(tool), "{text}"); + } + } +} + +#[test] +fn a_special_file_says_it_is_neither_a_file_nor_a_directory() { + assert_eq!( + neither_file_nor_directory("/dev/null"), + "Path is neither a file nor a directory: /dev/null" + ); +} diff --git a/crates/plugin-std/src/spec-tests.rs b/crates/plugin-std/src/spec-tests.rs index 4bca8158c..3f25110fa 100644 --- a/crates/plugin-std/src/spec-tests.rs +++ b/crates/plugin-std/src/spec-tests.rs @@ -5,13 +5,14 @@ use super::all; /// The wire name each id segment must carry, which is Cursor's name for /// the same tool. -const WIRE_NAMES: [(&str, &str); 6] = [ +const WIRE_NAMES: [(&str, &str); 7] = [ ("read", "Read"), ("write", "Write"), ("replace", "StrReplace"), ("rm", "Delete"), ("glob", "Glob"), ("grep", "Grep"), + ("dir", "ListDir"), ]; #[test] @@ -87,6 +88,103 @@ fn every_required_parameter_is_one_the_schema_declares_and_describes() { } } +#[test] +fn the_read_description_holds_the_limits_the_code_uses_and_no_unfilled_placeholder() { + let definitions = all().expect("every embedded definition parses"); + let read = definitions + .iter() + .find(|definition| definition.segment == "read") + .expect("the read definition exists"); + let text = &read.description; + assert!(text.contains("100000"), "the byte cap is named: {text}"); + assert!( + text.contains("16777216"), + "the file ceiling is named: {text}" + ); + assert!(!text.contains("cat -n"), "the cat -n line is gone: {text}"); + assert!( + !text.contains("2000 characters"), + "the 2000-character line is gone: {text}" + ); + for placeholder in ["{byte_cap}", "{file_ceiling}"] { + assert!( + !text.contains(placeholder), + "{placeholder} is filled in: {text}" + ); + } +} + +#[test] +fn the_dir_description_holds_the_body_cap_and_no_unfilled_placeholder() { + let definitions = all().expect("every embedded definition parses"); + let dir = definitions + .iter() + .find(|definition| definition.segment == "dir") + .expect("the dir definition exists"); + let text = &dir.description; + assert!(text.contains("15000"), "the body cap is named: {text}"); + assert!( + !text.contains("{body_cap}"), + "{{body_cap}} is filled in: {text}" + ); +} + +#[test] +fn the_dir_schema_requires_the_target_and_takes_an_array_of_strings_for_the_ignores() { + let definitions = all().expect("every embedded definition parses"); + let dir = definitions + .iter() + .find(|definition| definition.segment == "dir") + .expect("the dir definition exists"); + let properties = &dir.parameters["properties"]; + assert_eq!(properties["target_directory"]["type"], "string"); + assert_eq!(properties["ignore_globs"]["type"], "array"); + assert_eq!(properties["ignore_globs"]["items"]["type"], "string"); + assert_eq!( + dir.parameters["required"], + serde_json::json!(["target_directory"]) + ); +} + +#[test] +fn the_dir_description_states_the_behavior_the_tool_has() { + let definitions = all().expect("every embedded definition parses"); + let dir = definitions + .iter() + .find(|definition| definition.segment == "dir") + .expect("the dir definition exists"); + let text = &dir.description; + for phrase in [ + "breadth first", + "`.git`", + "dotfile", + "link", + "ignore_globs", + "any depth", + "byte order", + ] { + assert!( + text.contains(phrase), + "the description names {phrase}: {text}" + ); + } +} + +#[test] +fn the_read_schema_takes_integers_for_offset_and_limit() { + let definitions = all().expect("every embedded definition parses"); + let read = definitions + .iter() + .find(|definition| definition.segment == "read") + .expect("the read definition exists"); + for name in ["offset", "limit"] { + assert_eq!( + read.parameters["properties"][name]["type"], "integer", + "{name} is an integer" + ); + } +} + #[test] fn no_definition_names_behavior_this_build_does_not_have() { for definition in all().expect("every embedded definition parses") { @@ -100,3 +198,133 @@ fn no_definition_names_behavior_this_build_does_not_have() { } } } + +/// The `grep` definition. +fn grep_definition() -> &'static super::Definition { + all() + .expect("every embedded definition parses") + .iter() + .find(|definition| definition.segment == "grep") + .expect("the grep definition exists") +} + +#[test] +fn the_grep_description_names_the_dialect_the_limits_and_what_is_skipped() { + let text = &grep_definition().description; + for phrase in [ + "Rust regex", + "literal", + "15000", + "16777216", + "100000", + "skipped", + ".gitignore", + "1000 bytes", + "absolute", + "N:", + "N-", + "Read shows", + "no look-around", + "no back-references", + ] { + assert!( + text.contains(phrase), + "the description names {phrase}: {text}" + ); + } + for absent in ["ripgrep", "rg ", "several thousand", "Shell"] { + assert!( + !text.contains(absent), + "the description no longer names {absent}: {text}" + ); + } + for placeholder in ["{body_cap}", "{file_ceiling}"] { + assert!( + !text.contains(placeholder), + "{placeholder} is filled in: {text}" + ); + } +} + +#[test] +fn the_grep_schema_has_the_thirteen_parameters_in_order_with_their_types() { + let properties = grep_definition().parameters["properties"] + .as_object() + .expect("the schema has properties"); + // The parsed object is sorted by name, so the order the file declares + // them in is read from the file's own text. + let declared = [ + "pattern", + "path", + "glob", + "output_mode", + "-B", + "-A", + "-C", + "-i", + "type", + "head_limit", + "offset", + "multiline", + "literal", + ]; + let source = include_str!("../schemas/grep.json"); + let positions: Vec = declared + .iter() + .map(|name| { + source + .find(&format!("\n \"{name}\": {{")) + .unwrap_or_else(|| panic!("{name} is declared")) + }) + .collect(); + assert!( + positions.windows(2).all(|pair| pair[0] < pair[1]), + "the parameters are declared in order: {positions:?}" + ); + assert_eq!(properties.len(), declared.len()); + for name in declared { + assert!(properties.contains_key(name), "{name}"); + } + assert!(!properties.contains_key("-n")); + assert!(!properties.contains_key("context")); + for name in ["pattern", "path", "glob", "type", "output_mode"] { + assert_eq!(properties[name]["type"], "string", "{name}"); + } + for name in ["-B", "-A", "-C", "head_limit", "offset"] { + assert_eq!(properties[name]["type"], "number", "{name}"); + } + for name in ["-i", "multiline", "literal"] { + assert_eq!(properties[name]["type"], "boolean", "{name}"); + } + assert_eq!( + properties["output_mode"]["enum"], + serde_json::json!(["content", "files_with_matches", "count"]) + ); + assert_eq!(properties["output_mode"]["default"], "content"); + assert_eq!(properties["literal"]["default"], false); + assert_eq!(properties["head_limit"]["minimum"], 0); + assert!(properties["head_limit"].get("default").is_none()); + assert_eq!( + grep_definition().parameters["required"], + serde_json::json!(["pattern"]) + ); +} + +#[test] +fn the_grep_glob_and_path_texts_say_what_they_take() { + let properties = &grep_definition().parameters["properties"]; + let glob = properties["glob"]["description"] + .as_str() + .expect("glob has a description"); + assert!(glob.contains("\"*.js\""), "{glob}"); + assert!(glob.contains("\"**/*.tsx\""), "{glob}"); + assert!(!glob.contains('{'), "{glob}"); + let path = properties["path"]["description"] + .as_str() + .expect("path has a description"); + assert!(path.contains("absolute"), "{path}"); + let head_limit = properties["head_limit"]["description"] + .as_str() + .expect("head_limit has a description"); + assert!(head_limit.contains("0 for no limit"), "{head_limit}"); +} diff --git a/crates/plugin-std/src/spec.rs b/crates/plugin-std/src/spec.rs index c68b6233d..70a2a006e 100644 --- a/crates/plugin-std/src/spec.rs +++ b/crates/plugin-std/src/spec.rs @@ -1,10 +1,11 @@ -//! The six tool definitions this crate ships, each an embedded JSON file. +//! The seven tool definitions this crate ships, each an embedded JSON file. //! //! A file holds the name the model calls the tool by, the description the //! model reads, and the JSON Schema its arguments must match. The names, //! descriptions, and schemas are Cursor's own, with the lines that -//! describe behavior this Pack does not have removed; `schemas/EDITS.md` -//! records each of those edits. +//! describe behavior this Pack does not have removed, except for +//! `dir.json`, which is this crate's own; `schemas/EDITS.md` records each +//! of those edits and why `dir.json` is new. use std::sync::LazyLock; @@ -38,13 +39,14 @@ struct File { } /// Each tool's id segment beside the source of the file it reads. -const SOURCES: [(&str, &str); 6] = [ +const SOURCES: [(&str, &str); 7] = [ ("read", include_str!("../schemas/read.json")), ("write", include_str!("../schemas/write.json")), ("replace", include_str!("../schemas/replace.json")), ("rm", include_str!("../schemas/rm.json")), ("glob", include_str!("../schemas/glob.json")), ("grep", include_str!("../schemas/grep.json")), + ("dir", include_str!("../schemas/dir.json")), ]; /// The parsed definitions, one per [`SOURCES`] entry in order. @@ -62,13 +64,31 @@ static DEFINITIONS: LazyLock, ToolError>> = LazyLock::new Ok(Definition { segment, wire_name: file.wire_name, - description: file.description, + description: fill(segment, &file.description), parameters: file.parameters, }) }) .collect() }); +/// `description` with the tool's `{...}` placeholders filled from the +/// constants its code uses, so the text the model reads and the limit the +/// code enforces cannot drift apart. +/// +/// A tool without placeholders keeps its text as it is. +fn fill(segment: &str, description: &str) -> String { + match segment { + "read" => description + .replace("{byte_cap}", &crate::read::BYTE_CAP.to_string()) + .replace("{file_ceiling}", &crate::text::MAX_FILE_BYTES.to_string()), + "dir" => description.replace("{body_cap}", &crate::text::BODY_CAP.to_string()), + "grep" => description + .replace("{body_cap}", &crate::text::BODY_CAP.to_string()) + .replace("{file_ceiling}", &crate::text::MAX_FILE_BYTES.to_string()), + _ => description.to_owned(), + } +} + /// Every definition, in [`SOURCES`] order. /// /// # Errors diff --git a/crates/plugin-std/src/text.rs b/crates/plugin-std/src/text.rs new file mode 100644 index 000000000..6fcaf3840 --- /dev/null +++ b/crates/plugin-std/src/text.rs @@ -0,0 +1,129 @@ +//! How the file tools turn a file's bytes into lines of text. +//! +//! Read and Grep both go through this module, so a line number Grep +//! reports is the number Read prints. Nothing here returns an error to +//! the model for an encoding: bytes that are not valid text are replaced +//! and counted, and a file that is not text at all is told apart by +//! [`sniff_binary`]. + +/// The largest file, in bytes, a file tool reads: 16 MiB. +pub(crate) const MAX_FILE_BYTES: u64 = 16_777_216; + +/// The most bytes of tree or match lines a listing or search returns: +/// 15,000. ListDir and Grep share it. +pub(crate) const BODY_CAP: usize = 15_000; + +/// How many leading bytes [`sniff_binary`] looks at. +const SNIFF_BYTES: usize = 8_192; + +/// A file's bytes decoded to text. +#[derive(Debug, PartialEq, Eq)] +pub(crate) struct Decoded { + /// The text, without a leading byte order mark. + pub(crate) text: String, + /// `UTF-16LE` or `UTF-16BE` when the file started with that mark, + /// and `None` for anything else. + pub(crate) utf16: Option<&'static str>, + /// How many invalid UTF-8 sequences were shown as U+FFFD. It is `0` + /// for a UTF-16 file. + pub(crate) invalid: usize, +} + +/// Decodes `bytes` to text. +/// +/// A leading UTF-8 mark (`EF BB BF`) is dropped. A file that starts with +/// `FF FE` or `FE FF` is UTF-16, little-endian or big-endian: the mark is +/// dropped and the rest is decoded lossily, and a trailing odd byte is +/// ignored. Anything else is decoded as UTF-8, and each invalid chunk +/// becomes one U+FFFD and one count. No other encoding is guessed. +pub(crate) fn decode(bytes: &[u8]) -> Decoded { + if let Some(rest) = bytes.strip_prefix(&[0xFF, 0xFE]) { + return utf16(rest, "UTF-16LE", u16::from_le_bytes); + } + if let Some(rest) = bytes.strip_prefix(&[0xFE, 0xFF]) { + return utf16(rest, "UTF-16BE", u16::from_be_bytes); + } + let bytes = bytes.strip_prefix(&[0xEF, 0xBB, 0xBF]).unwrap_or(bytes); + let mut text = String::with_capacity(bytes.len()); + let mut invalid = 0; + for chunk in bytes.utf8_chunks() { + text.push_str(chunk.valid()); + if !chunk.invalid().is_empty() { + text.push(char::REPLACEMENT_CHARACTER); + invalid += 1; + } + } + Decoded { + text, + utf16: None, + invalid, + } +} + +/// Decodes the UTF-16 `bytes` that follow a mark, reading each unit with +/// `unit`, and names the byte order `name`. +fn utf16(bytes: &[u8], name: &'static str, unit: fn([u8; 2]) -> u16) -> Decoded { + let (pairs, _odd_byte) = bytes.as_chunks::<2>(); + let units: Vec = pairs.iter().map(|pair| unit(*pair)).collect(); + Decoded { + text: String::from_utf16_lossy(&units), + utf16: Some(name), + invalid: 0, + } +} + +/// Whether `bytes` look like binary data rather than text. +/// +/// Only the first 8,192 bytes count. They are binary when any of them is +/// NUL, or when at least 5% of them are control bytes other than tab, +/// line feed, and carriage return. A file that starts with a UTF-16 mark +/// is text whatever it holds, because UTF-16 text has NUL bytes by +/// design; UTF-16 without a mark is binary. +pub(crate) fn sniff_binary(bytes: &[u8]) -> bool { + if bytes.starts_with(&[0xFF, 0xFE]) || bytes.starts_with(&[0xFE, 0xFF]) { + return false; + } + let sample = bytes.get(..SNIFF_BYTES).unwrap_or(bytes); + if sample.contains(&0) { + return true; + } + let controls = sample + .iter() + .filter(|byte| byte.is_ascii_control() && !matches!(**byte, b'\t' | b'\n' | b'\r')) + .count(); + !sample.is_empty() && controls * 20 >= sample.len() +} + +/// Splits `text` into lines on `\n`. +/// +/// The empty piece after a final newline is not a line, and one trailing +/// `\r` is stripped from each line. So `a\nb\n` and `a\nb` are both two +/// lines, an empty text has none, and a lone `\r` inside a line stays in +/// it. +pub(crate) fn split_lines(text: &str) -> Vec<&str> { + text.split_terminator('\n') + .map(|line| line.strip_suffix('\r').unwrap_or(line)) + .collect() +} + +/// The part of `text` from byte `start` to byte `end`, cut on character +/// boundaries. +/// +/// `start` is rounded up and `end` down to the nearest boundary, and both +/// are clamped to the length of `text`. This is the only function that +/// shortens text inside a line, so no caller slices on a raw byte index. +pub(crate) fn clip_to_boundary(text: &str, start: usize, end: usize) -> &str { + let length = text.len(); + let mut start = start.min(length); + let mut end = end.min(length); + while !text.is_char_boundary(start) { + start += 1; + } + while !text.is_char_boundary(end) { + end -= 1; + } + text.get(start..end).unwrap_or("") +} + +#[cfg(test)] +mod tests; diff --git a/crates/plugin-std/src/text/tests.rs b/crates/plugin-std/src/text/tests.rs new file mode 100644 index 000000000..5bb5b06b0 --- /dev/null +++ b/crates/plugin-std/src/text/tests.rs @@ -0,0 +1,285 @@ +//! Unit tests for the text module: the decode ladder, the binary sniff, +//! the line splitter, and the boundary clip. + +use super::{BODY_CAP, MAX_FILE_BYTES, clip_to_boundary, decode, sniff_binary, split_lines}; + +/// The UTF-16 little-endian bytes of `text`, behind the `FF FE` mark. +fn utf16_le(text: &str) -> Vec { + let mut bytes = vec![0xFF, 0xFE]; + for unit in text.encode_utf16() { + bytes.extend_from_slice(&unit.to_le_bytes()); + } + bytes +} + +/// The UTF-16 big-endian bytes of `text`, behind the `FE FF` mark. +fn utf16_be(text: &str) -> Vec { + let mut bytes = vec![0xFE, 0xFF]; + for unit in text.encode_utf16() { + bytes.extend_from_slice(&unit.to_be_bytes()); + } + bytes +} + +#[test] +fn the_file_ceiling_is_sixteen_mebibytes() { + assert_eq!(MAX_FILE_BYTES, 16 * 1024 * 1024); + assert_eq!(MAX_FILE_BYTES, 16_777_216); +} + +#[test] +fn the_body_cap_the_listing_and_search_tools_share_is_fifteen_thousand_bytes() { + assert_eq!(BODY_CAP, 15_000); +} + +#[test] +fn plain_utf8_decodes_unchanged_with_no_note() { + let decoded = decode("hello\nwörld €\n".as_bytes()); + assert_eq!(decoded.text, "hello\nwörld €\n"); + assert_eq!(decoded.utf16, None); + assert_eq!(decoded.invalid, 0); +} + +#[test] +fn a_leading_utf8_mark_is_dropped_and_not_reported() { + let decoded = decode(b"\xEF\xBB\xBFhi\n"); + assert_eq!(decoded.text, "hi\n"); + assert_eq!(decoded.utf16, None); + assert_eq!(decoded.invalid, 0); +} + +#[test] +fn a_mark_that_is_not_leading_stays_in_the_text() { + let decoded = decode(b"a\xEF\xBB\xBFb"); + assert_eq!(decoded.text, "a\u{FEFF}b"); +} + +#[test] +fn utf16_little_endian_with_a_mark_is_decoded_and_named() { + let decoded = decode(&utf16_le("hi\nthere\n")); + assert_eq!(decoded.text, "hi\nthere\n"); + assert_eq!(decoded.utf16, Some("UTF-16LE")); + assert_eq!(decoded.invalid, 0); +} + +#[test] +fn utf16_big_endian_with_a_mark_is_decoded_and_named() { + let decoded = decode(&utf16_be("hi\nthere\n")); + assert_eq!(decoded.text, "hi\nthere\n"); + assert_eq!(decoded.utf16, Some("UTF-16BE")); + assert_eq!(decoded.invalid, 0); +} + +#[test] +fn utf16_decodes_a_surrogate_pair_and_replaces_a_lone_surrogate() { + let decoded = decode(&utf16_le("a\u{1F600}b")); + assert_eq!(decoded.text, "a\u{1F600}b"); + let lone = [0xFF, 0xFE, 0x61, 0x00, 0x00, 0xD8, 0x62, 0x00]; + let decoded = decode(&lone); + assert_eq!(decoded.text, "a\u{FFFD}b"); + assert_eq!(decoded.utf16, Some("UTF-16LE")); + assert_eq!(decoded.invalid, 0, "the count is for UTF-8 sequences only"); +} + +#[test] +fn utf16_drops_one_trailing_byte_that_does_not_make_a_unit() { + let decoded = decode(&[0xFF, 0xFE, 0x61, 0x00, 0x62]); + assert_eq!(decoded.text, "a"); + assert_eq!(decoded.utf16, Some("UTF-16LE")); +} + +#[test] +fn a_file_that_is_only_a_mark_decodes_to_nothing() { + assert_eq!(decode(b"\xEF\xBB\xBF").text, ""); + assert_eq!(decode(&[0xFF, 0xFE]).text, ""); + assert_eq!(decode(&[0xFE, 0xFF]).text, ""); + assert_eq!(decode(b"").text, ""); +} + +#[test] +fn an_invalid_byte_becomes_one_replacement_character_and_one_count() { + let decoded = decode(&[0x61, 0xFF, 0x62, 0x0A]); + assert_eq!(decoded.text, "a\u{FFFD}b\n"); + assert_eq!(decoded.utf16, None); + assert_eq!(decoded.invalid, 1); +} + +#[test] +fn a_file_that_ends_mid_sequence_counts_one() { + let decoded = decode(&[0x61, 0xE2, 0x82]); + assert_eq!(decoded.text, "a\u{FFFD}"); + assert_eq!(decoded.invalid, 1); +} + +#[test] +fn separate_invalid_chunks_each_count_once() { + let decoded = decode(&[0xFF, 0x61, 0xC0, 0xC1, 0x62, 0xE2, 0x82]); + assert_eq!(decoded.invalid, 4, "FF, C0, C1, and the cut E2 82"); + assert_eq!(decoded.text.matches('\u{FFFD}').count(), 4); +} + +#[test] +fn bytes_after_a_utf8_mark_are_counted_too() { + let decoded = decode(&[0xEF, 0xBB, 0xBF, 0x61, 0xFF]); + assert_eq!(decoded.text, "a\u{FFFD}"); + assert_eq!(decoded.invalid, 1); +} + +#[test] +fn ordinary_text_is_not_binary() { + assert!(!sniff_binary(b"fn main() {\n\tprintln!(\"hi\");\r\n}\n")); + assert!(!sniff_binary("héllo wörld €\n".as_bytes())); + assert!(!sniff_binary(b"")); +} + +#[test] +fn a_nul_in_the_first_8192_bytes_is_binary() { + assert!(sniff_binary(b"\0abc")); + let mut bytes = vec![b'a'; 9_000]; + bytes[8_191] = 0; + assert!(sniff_binary(&bytes), "index 8191 is inside the sample"); + let mut bytes = vec![b'a'; 9_000]; + bytes[8_192] = 0; + assert!(!sniff_binary(&bytes), "index 8192 is outside the sample"); +} + +#[test] +fn the_sniff_ignores_everything_past_the_sample() { + let mut bytes = vec![b'a'; 8_192]; + bytes.extend(vec![0x01; 8_192]); + assert!(!sniff_binary(&bytes)); +} + +#[test] +fn five_percent_control_bytes_is_binary_and_less_is_not() { + let sample = |controls: usize| { + let mut bytes = vec![b'a'; 1_000 - controls]; + bytes.extend(vec![0x01; controls]); + bytes + }; + assert!(sniff_binary(&sample(50)), "50 of 1000 is 5%"); + assert!(!sniff_binary(&sample(49)), "49 of 1000 is 4.9%"); + assert!(sniff_binary(&sample(1_000))); +} + +#[test] +fn a_delete_byte_counts_as_a_control_byte() { + let mut bytes = vec![b'a'; 950]; + bytes.extend(vec![0x7F; 50]); + assert!(sniff_binary(&bytes)); +} + +#[test] +fn tab_line_feed_and_carriage_return_never_count() { + let bytes: Vec = (0..1_000).map(|n| b"\t\n\r"[n % 3]).collect(); + assert!(!sniff_binary(&bytes)); +} + +#[test] +fn a_file_with_a_utf16_mark_skips_the_sniff_and_one_without_is_binary() { + assert!(!sniff_binary(&utf16_le("hello\n"))); + assert!(!sniff_binary(&utf16_be("hello\n"))); + assert!( + sniff_binary(&[b'h', 0, b'i', 0]), + "UTF-16 without a mark holds NUL bytes" + ); +} + +#[test] +fn a_final_newline_does_not_make_an_extra_line() { + assert_eq!(split_lines("a\nb"), ["a", "b"]); + assert_eq!(split_lines("a\nb\n"), ["a", "b"]); +} + +#[test] +fn an_empty_text_has_no_lines_and_a_lone_newline_has_one() { + assert!(split_lines("").is_empty()); + assert_eq!(split_lines("\n"), [""]); + assert_eq!(split_lines("a\n\nb\n"), ["a", "", "b"]); +} + +#[test] +fn one_trailing_carriage_return_is_stripped_from_each_line() { + assert_eq!(split_lines("a\r\nb\r\n"), ["a", "b"]); + assert_eq!(split_lines("a\r\nb\r"), ["a", "b"]); + assert_eq!(split_lines("\r\n"), [""]); + assert_eq!(split_lines("a\r\r\nb"), ["a\r", "b"]); +} + +#[test] +fn a_lone_carriage_return_inside_a_line_is_kept() { + assert_eq!( + split_lines("first\rNEEDLE\rthird"), + ["first\rNEEDLE\rthird"] + ); +} + +/// The text of one 1-byte, one 2-byte, one 3-byte, and one 4-byte +/// character, twice. +const MIXED: &str = "aé€😀aé€😀"; + +/// The smallest character boundary at or after `at`, within `text`. +fn boundary_up(text: &str, at: usize) -> usize { + (at.min(text.len())..=text.len()) + .find(|&n| text.is_char_boundary(n)) + .expect("the end of a string is a boundary") +} + +/// The largest character boundary at or before `at`, within `text`. +fn boundary_down(text: &str, at: usize) -> usize { + (0..=at.min(text.len())) + .rev() + .find(|&n| text.is_char_boundary(n)) + .expect("the start of a string is a boundary") +} + +#[test] +fn clip_gives_the_exact_range_when_both_ends_are_boundaries() { + assert_eq!(clip_to_boundary("hello", 1, 4), "ell"); + assert_eq!(clip_to_boundary("hello", 0, 5), "hello"); + assert_eq!(clip_to_boundary("hello", 2, 2), ""); +} + +#[test] +fn clip_rounds_the_start_up_and_the_end_down_over_every_range() { + for text in ["a", "é", "€", "😀", "aé€😀", MIXED] { + let len = text.len(); + for start in 0..=len + 2 { + for end in 0..=len + 2 { + let clipped = clip_to_boundary(text, start, end); + let low = boundary_up(text, start); + let high = boundary_down(text, end); + let expected = if low >= high { "" } else { &text[low..high] }; + assert_eq!(clipped, expected, "{text:?} from {start} to {end}"); + if !clipped.is_empty() { + let at = clipped.as_ptr().addr() - text.as_ptr().addr(); + assert!( + at >= start && at + clipped.len() <= end.min(len), + "{text:?} from {start} to {end} gave bytes {at}..{}", + at + clipped.len() + ); + } + } + } + } +} + +#[test] +fn clip_clamps_both_ends_to_the_length_and_a_reversed_range_is_empty() { + assert_eq!(clip_to_boundary("abc", 1, 100), "bc"); + assert_eq!(clip_to_boundary("abc", 100, 200), ""); + assert_eq!(clip_to_boundary("abc", 3, 1), ""); + assert_eq!(clip_to_boundary("", 0, 5), ""); +} + +#[test] +fn a_prefix_cut_never_splits_a_character() { + for width in 0..=MIXED.len() { + let kept = clip_to_boundary(MIXED, 0, width); + assert!(MIXED.starts_with(kept), "{width}"); + assert!(kept.len() <= width, "{width}"); + assert!(width - kept.len() < 4, "the cut stops within one character"); + } + assert_eq!(clip_to_boundary("€€", 0, 5), "€"); + assert_eq!(clip_to_boundary("€€", 0, 2), ""); +} diff --git a/crates/plugin-std/tests/it/grep.rs b/crates/plugin-std/tests/it/grep.rs new file mode 100644 index 000000000..6c2c8b817 --- /dev/null +++ b/crates/plugin-std/tests/it/grep.rs @@ -0,0 +1,90 @@ +//! Grep's cooperative yield, seen from the Pack's own future: a search over +//! many files hands control back to the executor before it completes, and +//! dropping it at a pause leaves the filesystem usable. + +use std::future::Future; +use std::pin::pin; +use std::task::{Context, Poll, Waker}; + +use runvil_plugin::testing::TestCall; +use runvil_plugin::{OutputTrust, ToolError, ToolOutput}; +use serde_json::{Value, json}; + +use crate::tools::{built, id}; + +/// How many files the yield check seeds. +const FILES: usize = 1000; + +/// A call with `FILES` small files, each holding one matching line. +fn seeded_call() -> TestCall { + let call = TestCall::new(id("grep")); + for index in 0..FILES { + call.context() + .access() + .write(&format!("/f{index:04}.txt"), b"x\n") + .expect("a seeded file is written"); + } + call +} + +/// The search arguments the yield check uses: one count row per file. +fn args() -> Value { + json!({ "pattern": "x", "output_mode": "count" }) +} + +#[test] +fn a_search_over_a_thousand_files_yields_to_the_executor_before_it_completes() { + let pack = built(); + let call = seeded_call(); + let mut future = pin!(pack.call(call.context(), args())); + let mut cx = Context::from_waker(Waker::noop()); + let mut pending = 0; + let result: Result = loop { + match future.as_mut().poll(&mut cx) { + Poll::Ready(result) => break result, + Poll::Pending => pending += 1, + } + }; + assert!( + pending >= 31, + "a pause after each 32 of {FILES} files, and the search paused {pending} times" + ); + let output = result.expect("the search completes with its result"); + assert_eq!(output.trust(), OutputTrust::Untrusted); + let rows: Vec<&str> = output.text().lines().collect(); + assert_eq!( + rows.len(), + FILES, + "every file has its row and there is no notice" + ); + assert_eq!(rows.first(), Some(&"/f0000.txt:1")); + assert_eq!(rows.last(), Some(&"/f0999.txt:1")); +} + +#[test] +fn dropping_a_search_after_its_first_pause_is_safe_and_the_access_still_works() { + let pack = built(); + let call = seeded_call(); + { + let mut dropped = pin!(pack.call(call.context(), args())); + let mut cx = Context::from_waker(Waker::noop()); + assert!( + dropped.as_mut().poll(&mut cx).is_pending(), + "the search pauses before it can finish" + ); + } + let bytes = call + .context() + .access() + .read("/f0000.txt") + .expect("the access still reads after the drop"); + assert_eq!(bytes, b"x\n"); + let mut again = pin!(pack.call(call.context(), args())); + let mut cx = Context::from_waker(Waker::noop()); + let output = loop { + if let Poll::Ready(result) = again.as_mut().poll(&mut cx) { + break result.expect("a fresh search runs to its result"); + } + }; + assert_eq!(output.text().lines().count(), FILES); +} diff --git a/crates/plugin-std/tests/it/main.rs b/crates/plugin-std/tests/it/main.rs index cb6481d6c..f67fb87c8 100644 --- a/crates/plugin-std/tests/it/main.rs +++ b/crates/plugin-std/tests/it/main.rs @@ -6,4 +6,5 @@ reason = "test helpers panic on setup failure, which is the desired behavior" )] +mod grep; mod tools; diff --git a/crates/plugin-std/tests/it/tools.rs b/crates/plugin-std/tests/it/tools.rs index 21a02897b..d6d895fc2 100644 --- a/crates/plugin-std/tests/it/tools.rs +++ b/crates/plugin-std/tests/it/tools.rs @@ -1,5 +1,5 @@ //! The `std` Pack through the crate's `install`: the one Pack it inserts -//! with its config, the six tools `construct` names under the Pack id it +//! with its config, the seven tools `construct` names under the Pack id it //! receives, and what a call answers. use std::future::Future; @@ -56,7 +56,7 @@ fn construct(name: &str, config: Value) -> Result, ToolError> { } /// The Pack built under `std`. -fn built() -> Arc { +pub(crate) fn built() -> Arc { let Ok(pack) = construct("std", Value::Null) else { panic!("construct succeeds with no configuration"); }; @@ -64,14 +64,30 @@ fn built() -> Arc { } /// The tool id `segment` under the Pack id `std`. -fn id(segment: &str) -> ToolId { +pub(crate) fn id(segment: &str) -> ToolId { ToolId::parse(&format!("std/{segment}")).expect("a two-segment tool id parses") } /// Calls the tool `segment` with `args`. fn call(segment: &str, args: Value) -> Result { + call_seeded(segment, &[], args) +} + +/// Calls the tool `segment` with `args` after writing each of `files`, a +/// path beside its bytes, through the call's own access. +fn call_seeded( + segment: &str, + files: &[(&str, &[u8])], + args: Value, +) -> Result { let pack = built(); let call = TestCall::new(id(segment)); + for (path, bytes) in files { + call.context() + .access() + .write(path, bytes) + .expect("a seeded file is written"); + } block_on(pack.call(call.context(), args)) } @@ -111,7 +127,7 @@ fn install_returns_the_registrys_refusal() { } #[test] -fn construct_names_six_tools_under_the_pack_id_it_receives() { +fn construct_names_seven_tools_under_the_pack_id_it_receives() { let Ok(pack) = construct("files", Value::Null) else { panic!("construct succeeds with no configuration"); }; @@ -128,7 +144,8 @@ fn construct_names_six_tools_under_the_pack_id_it_receives() { "files/replace", "files/rm", "files/glob", - "files/grep" + "files/grep", + "files/dir" ] ); } @@ -150,6 +167,7 @@ fn every_tool_carries_cursors_name_for_it_as_its_wire_name() { ("std/rm".to_owned(), Some("Delete".to_owned())), ("std/glob".to_owned(), Some("Glob".to_owned())), ("std/grep".to_owned(), Some("Grep".to_owned())), + ("std/dir".to_owned(), Some("ListDir".to_owned())), ] ); } @@ -181,23 +199,20 @@ fn construct_accepts_null_or_an_empty_object_and_refuses_any_other_configuration #[test] fn every_tool_answers_the_not_implemented_text_naming_itself() { for (segment, wire_name) in [ - ("read", "Read"), ("write", "Write"), ("replace", "StrReplace"), ("rm", "Delete"), ("glob", "Glob"), - ("grep", "Grep"), ] { let args = match segment { - "read" | "rm" => json!({ "path": "/notes.md" }), + "rm" => json!({ "path": "/notes.md" }), "write" => json!({ "path": "/notes.md", "contents": "hi" }), "replace" => json!({ "path": "/notes.md", "old_string": "a", "new_string": "b" }), - "glob" => json!({ "glob_pattern": "**/*.md" }), - _ => json!({ "pattern": "TODO" }), + _ => json!({ "glob_pattern": "**/*.md" }), }; let output = call(segment, args).expect("a matching call succeeds"); let expected = NOT_IMPLEMENTED.replace("", wire_name); @@ -258,3 +273,116 @@ fn a_call_of_a_tool_the_pack_lacks_fails_naming_it() { block_on(pack.call(unknown.context(), json!({}))).expect_err("std has no copy tool"); assert_eq!(error.to_string(), "std has no tool named copy"); } + +#[test] +fn read_answers_the_numbered_lines_of_a_seeded_file_as_untrusted_output() { + let output = call_seeded( + "read", + &[("/notes.md", b"hello\nworld\n")], + json!({ "path": "/notes.md" }), + ) + .expect("a seeded file reads"); + assert_eq!(output.text(), " 1|hello\n 2|world\n"); + assert_eq!(output.trust(), OutputTrust::Untrusted); +} + +#[test] +fn list_dir_answers_the_tree_of_a_seeded_directory_as_untrusted_output() { + let output = call_seeded( + "dir", + &[ + ("/proj/README.md", b"hello"), + ("/proj/src/lib.rs", b"fn main() {}"), + ("/proj/.git/HEAD", b"ref"), + ], + json!({ "target_directory": "/proj" }), + ) + .expect("a seeded directory lists"); + assert_eq!( + output.text(), + "/proj/\n - README.md\n - src/\n - lib.rs" + ); + assert_eq!(output.trust(), OutputTrust::Untrusted); +} + +#[test] +fn list_dir_of_a_missing_path_fails_with_the_does_not_exist_text() { + let error = + call("dir", json!({ "target_directory": "/nope" })).expect_err("the directory is missing"); + assert_eq!(error.to_string(), "Path does not exist: /nope"); +} + +#[test] +fn list_dir_leaves_out_what_ignore_globs_match() { + let output = call_seeded( + "dir", + &[("/proj/a.md", b"x"), ("/proj/b.log", b"y")], + json!({ "target_directory": "/proj", "ignore_globs": ["*.log"] }), + ) + .expect("a seeded directory lists"); + assert_eq!(output.text(), "/proj/\n - a.md"); +} + +#[test] +fn read_answers_file_is_empty_for_a_zero_byte_file() { + let output = call_seeded( + "read", + &[("/empty.md", b"")], + json!({ "path": "/empty.md" }), + ) + .expect("an empty file reads"); + assert_eq!(output.text(), "File is empty."); + assert_eq!(output.trust(), OutputTrust::Untrusted); +} + +#[test] +fn read_of_a_missing_path_fails_with_the_not_found_text() { + let error = call("read", json!({ "path": "/nope.md" })).expect_err("the file is missing"); + assert_eq!(error.to_string(), "File not found: /nope.md"); +} + +#[test] +fn grep_answers_the_matching_lines_of_a_seeded_tree_as_untrusted_output() { + let output = call_seeded( + "grep", + &[ + ( + "/src/a.ts", + b"// one\n// two\nconst alpha = 1\nlet beta = 2\n// five\n// six\n// seven\n\ + // eight\nconst gamma = 3\nexport default gamma\n", + ), + ("/src/b.ts", b"const first = 0\n"), + ], + json!({ "pattern": "const", "-A": 1 }), + ) + .expect("a seeded tree searches"); + assert_eq!( + output.text(), + "/src/a.ts\n 3:const alpha = 1\n 4-let beta = 2\n --\n 9:const gamma = 3\n \ + 10-export default gamma\n\n/src/b.ts\n 1:const first = 0" + ); + assert_eq!(output.trust(), OutputTrust::Untrusted); +} + +#[test] +fn grep_of_a_missing_path_fails_with_the_does_not_exist_text() { + let error = + call("grep", json!({ "pattern": "x", "path": "/nope" })).expect_err("the path is missing"); + assert_eq!(error.to_string(), "Path does not exist: /nope"); +} + +#[test] +fn grep_refuses_an_undeclared_key_by_name_while_the_stubs_let_it_through() { + let error = call("grep", json!({ "pattern": "x", "context": 3 })) + .expect_err("context is not a parameter"); + assert_eq!(error.kind(), ToolErrorKind::InvalidArguments); + assert!( + error + .to_string() + .starts_with("std_Grep: `context` is not a parameter of this tool."), + "{error}" + ); + let stub = call("glob", json!({ "glob_pattern": "*.md", "extra": 1 })) + .expect("a stub ignores an undeclared key"); + assert_eq!(stub.text(), NOT_IMPLEMENTED.replace("", "Glob")); +} diff --git a/crates/runvil-internal/engine/README.md b/crates/runvil-internal/engine/README.md index 19bfa6ecd..6d9e6c797 100644 --- a/crates/runvil-internal/engine/README.md +++ b/crates/runvil-internal/engine/README.md @@ -1,3 +1,3 @@ # Engine executor -This crate is the Engine's executor: it runs a parsed prompt as a deterministic state machine. Each model round, Pack render, Pack tool call, `store` operation, and timer leaves the executor as an effect for the Harness to perform and answer, and every boundary is reported as an event. Adhoc tool calls, `GetDynamicTools` calls, refused discovery calls, and the store calls a Pack prelude makes while it installs run inside the Engine and are reported only as events, and the run's start probe reads the store directly. Given the same prompt, arguments, and `RunContext` inputs, its seed and start time included, and the same answers delivered in the same order at the same steps, a run produces the same effects and events. +This crate is the Engine's executor: it runs a parsed prompt as a deterministic state machine. Each model round, Pack render, Pack tool call, `store` operation, and timer leaves the executor as an effect for the Harness to perform and answer, and every boundary is reported as an event. Adhoc tool calls, `GetDynamicTools` calls, refused discovery calls, and the store calls a Pack prelude makes while it installs run inside the Engine and are reported only as events, and the run's start probe reads the store directly. Given the same prompt, arguments, and `RunConfig` inputs, its seed and start time included, and the same answers delivered in the same order at the same steps, a run produces the same effects and events. diff --git a/crates/runvil-internal/engine/benches/models_loop.rs b/crates/runvil-internal/engine/benches/models_loop.rs index b525d8bee..26c558b1b 100644 --- a/crates/runvil-internal/engine/benches/models_loop.rs +++ b/crates/runvil-internal/engine/benches/models_loop.rs @@ -19,15 +19,16 @@ reason = "bench setup panics on construction failure, which is the desired behavior" )] +use std::collections::BTreeMap; use std::num::NonZeroU32; use std::sync::Arc; use criterion::{Criterion, criterion_group, criterion_main}; use runvil_engine::test_support::scripted_chat::{ScriptedChat, ScriptedReply}; use runvil_engine::test_support::{drive, perform_locally}; -use runvil_engine::{Effect, EffectAnswer, Run, RunContext, RunResult}; +use runvil_engine::{Effect, EffectAnswer, Run, RunConfig, RunOutcome}; use runvil_parser::Prompt; -use runvil_types::models::{ModelCatalog, ModelId, ModelInfo, ThinkingMode}; +use runvil_types::models::{ModelCatalog, ModelId, ModelInfo, ModelRole, ThinkingMode}; use runvil_types::seed::RunSeed; use runvil_types::timestamp::Timestamp; @@ -61,11 +62,8 @@ fn bench_catalog(context: u32) -> ModelCatalog { } /// One section driving `models.loop` over its `messages` list. -const LOOP_PROMPT: &str = "---\nname: bench_loop\ndescription: d\nrunvil: 0\nmodels:\n writer: {}\n---\n\n\ +const LOOP_PROMPT: &str = "---\nname: bench_loop\ndescription: d\nrunvil: 0\n---\n\n\ # Bench\n\n\ - ```lua\n\ - models.default('writer')\n\ - ```\n\n\ ## Only\n\n\ ```lua\n\ messages:user('hi')\n\ @@ -80,18 +78,15 @@ fn parse_loop_prompt() -> Prompt { .expect("the bench prompt parses") } -/// Prepares and runs `prompt` on a `context`-token window, on the serial -/// driver: every effect is performed in process, and every round is -/// answered by `chat`. A prompt the prepare refuses ends with the refusal. -fn run_once(prompt: &Prompt, chat: &ScriptedChat, context: u32) -> RunResult { - let ctx = RunContext::new(EXECUTION, SEED, Timestamp::UNIX_EPOCH); +/// Runs `prompt` on a `context`-token window, on the serial driver: every +/// effect is performed in process, and every round is answered by `chat`. +/// A prompt the run's start checks refuse ends with the refusal. +fn run_once(prompt: &Prompt, chat: &ScriptedChat, context: u32) -> RunOutcome { let model = bench_catalog(context).models()[0].clone(); - let (ctx, requirements) = ctx.prepare(prompt, Some(model)); - if let Some(refusal) = requirements.refusal() { - return RunResult::Failure(refusal); - } + let ctx = RunConfig::new(EXECUTION, SEED, Timestamp::UNIX_EPOCH) + .with_models(BTreeMap::from([(ModelRole::Default, model)])); let run = Run::new(Arc::new(prompt.clone()), "", ctx); - let (result, _) = drive(run, |_, effect| { + let (outcome, _) = drive(run, |_, effect| { perform_locally(effect, &mut |round| { let Effect::Chat { binding, @@ -106,7 +101,7 @@ fn run_once(prompt: &Prompt, chat: &ScriptedChat, context: u32) -> RunResult { EffectAnswer::Chat(answer.map(Box::new)) }) }); - result + outcome } /// One `models.loop` turn end to end: parse is excluded, so the measurement @@ -116,10 +111,10 @@ fn models_loop(c: &mut Criterion) { let prompt = parse_loop_prompt(); c.bench_function("models_loop", |b| { b.iter(|| { - let result = run_once(&prompt, &chat, 131_072); + let outcome = run_once(&prompt, &chat, 131_072); assert!( - matches!(result, RunResult::Completed(_)), - "the loop bench run succeeds: {result:?}" + matches!(outcome, RunOutcome::Completed { .. }), + "the loop bench run succeeds: {outcome:?}" ); }); }); @@ -137,14 +132,13 @@ fn compactors_fail(c: &mut Criterion) { let prompt = parse_loop_prompt(); c.bench_function("compactors_fail", |b| { b.iter(|| { - let result = run_once(&prompt, &chat, 1); - let RunResult::Failure(error) = result else { + let RunOutcome::Failed { kind, message } = run_once(&prompt, &chat, 1) else { panic!("a one-token window must exhaust at the precheck"); }; assert_eq!( - error.kind(), + kind, runvil_engine::RunErrorKind::ContextExhausted, - "the default compactor is compactors.fail: {error:?}" + "the default compactor is compactors.fail: {message}" ); }); }); diff --git a/crates/runvil-internal/engine/src/error.rs b/crates/runvil-internal/engine/src/error.rs index 34d95dca9..04cc36ba6 100644 --- a/crates/runvil-internal/engine/src/error.rs +++ b/crates/runvil-internal/engine/src/error.rs @@ -1,21 +1,23 @@ //! The crate's internal error type. //! //! [`Error`] is a `pub(crate)` internal error type, never part of the public API. -//! The public boundary is [`crate::RunError`], and [`ParseError`](crate::parser::ParseError), +//! A run it ends reports its kind and message in +//! [`RunOutcome::Failed`](crate::RunOutcome::Failed), and +//! [`ParseError`](crate::parser::ParseError), //! [`CompletionError`](crate::model::CompletionError), and -//! [`runvil_types::tools::ToolError`] each reach it through `RunError`'s -//! classification, which classifies this internal type and keeps its cause. +//! [`runvil_types::tools::ToolError`] each reach that outcome through this +//! type, which classifies them and keeps their causes. //! The `From` bridges that let internal `?` reach this type live in `convert`. //! //! A variant that keeps an originating error renders that error's text in -//! its own `Display` and holds the error as a plain `cause` field behind -//! [`Error::originating_error`], never as a `#[source]`, so a printer that -//! walks the source chain shows the cause's text once. +//! its own `Display` and holds the error as a plain `cause` field, never as +//! a `#[source]`, so a printer that walks the source chain shows the +//! cause's text once. //! //! Three variants are `#[error(transparent)]` and so keep no `cause` field //! of their own: `Parse`, `Completion`, and `Substitution` hand their whole //! `Display` and `source` to the error they hold. That is the exception to -//! the field rule above, and it is why [`Error::originating_error`] answers +//! the field rule above, and it is why `Error::originating_error` answers //! `None` for `Completion` and `Substitution` and the parser's own cause for //! `Parse`. @@ -107,7 +109,7 @@ impl std::fmt::Display for DepthVia { /// failures. /// /// The model variant holds the broker's closed error whole, so no HTTP -/// client's error type leaks through the wrappers' `source()`. +/// client's error type leaks through its `source()`. #[derive(Debug, thiserror::Error)] pub(crate) enum Error { /// The prompt failed to parse: invalid frontmatter YAML or a @@ -115,7 +117,8 @@ pub(crate) enum Error { /// /// Holds the parser's [`ParseError`](crate::parser::ParseError) whole, so /// its stable kind, its location fields, and the YAML decode failure a - /// frontmatter failure keeps as its cause survive the public wrappers. + /// frontmatter failure keeps as its cause survive until the run's + /// failure message is built from them. #[error(transparent)] Parse(crate::parser::ParseError), @@ -124,7 +127,8 @@ pub(crate) enum Error { /// /// Holds the broker's [`CompletionError`](crate::model::CompletionError) /// whole, so the closed kind, the fixed message, the opt-in detail, and - /// the cause chain survive the public wrappers. The Engine branches on + /// the cause chain survive until the run's failure message is built + /// from them. The Engine branches on /// its kind only: a context overflow takes the provider overflow path /// and an empty reply takes the empty-answer path. #[error(transparent)] @@ -144,9 +148,9 @@ pub(crate) enum Error { /// It also holds the message of a raised error table whose kind needs /// structure the table does not hold (see `Error::from_raised`). /// `cause` holds the originating `mlua` error when there is one. The - /// message is the specific failure as a noun phrase; the public wrapper - /// classifies this as a Lua failure, so no redundant `lua error:` type - /// label is prepended. + /// message is the specific failure as a noun phrase; the run's failed + /// outcome classifies this as a Lua failure, so no redundant + /// `lua error:` type label is prepended. #[error("{message}")] Lua { /// The diagnostic, location-tagged when it maps to a prompt line. @@ -158,8 +162,7 @@ pub(crate) enum Error { /// Lua source was not syntactically valid at its prompt location. /// /// Keeps the originating `mlua` compile error as the cause beside the - /// location metadata, so the typed compiler diagnostic survives through - /// [`RunError::originating_error`](crate::RunError::originating_error) + /// location metadata, so the typed compiler diagnostic survives /// instead of being flattened into `message` alone. #[error("lua compilation error at {location} (line {source_line}): {message}")] LuaCompile { @@ -306,8 +309,25 @@ pub(crate) enum Error { ids: Vec, }, - /// A `models.infer` or `models.loop` round found no `models.use` - /// selection and no prompt-wide `models.default`. + /// A `tools.call` named a catalog tool that only a model may call. + /// + /// A Pack marks a tool model-only when a render gives its arguments a + /// script cannot know, so the tool stays in the catalog and a section + /// offers it, while only a model's call can supply arguments that fit. + #[error( + "tool {name:?} ({id}) is model-only: a render shapes its arguments for the model that \ + receives them, so a script cannot call it; offer it with tools.offer and let the model \ + call it" + )] + ModelOnlyToolCall { + /// The name the call tried to dispatch. + name: String, + /// The catalog tool's id. + id: String, + }, + + /// A `models.infer` or `models.loop` round found no current model, + /// because the caller supplied none. #[error("model binding required for section {section}")] ModelRequired { /// The name of the section whose round had no binding, or the @@ -357,7 +377,7 @@ pub(crate) enum Error { /// /// The Rust source position of the construction site is captured (via /// [`Error::internal`] or [`Error::internal_with_cause`], both - /// `#[track_caller]`) so [`crate::RunError::location`] can point at the + /// `#[track_caller]`) so the run's failure message can point at the /// broken invariant. A Lua-crate internal failure brings its own /// captured site. #[error("internal invariant violated: {message}{}", cause_suffix(.cause.as_ref()))] @@ -404,7 +424,7 @@ pub(crate) enum Error { /// A run-scoped store operation failed at the virtual filesystem layer, /// keeping the concrete [`runvil_vfs::VfsError`] as the cause so a - /// backend failure survives the public wrappers instead of being + /// backend failure survives as a typed error instead of being /// flattened to a string. The message is the model-facing rendering for /// the operation (or the probe text, which says when the run declares no /// store), kept beside the error because the error alone does not name @@ -437,6 +457,7 @@ pub(crate) enum Error { impl Error { /// Returns the originating error a cause-bearing variant keeps, whose /// text its `Display` already holds; `None` for every other variant. + #[cfg(test)] pub(crate) fn originating_error( &self, ) -> Option<&(dyn std::error::Error + Send + Sync + 'static)> { @@ -460,6 +481,7 @@ impl Error { | Error::TaskCancelled { .. } | Error::OutOfScopeToolCall { .. } | Error::UnboundToolCall { .. } + | Error::ModelOnlyToolCall { .. } | Error::ModelRequired { .. } | Error::UnsupportedVersion(_) | Error::LuaQuota { .. } diff --git a/crates/runvil-internal/engine/src/error/convert.rs b/crates/runvil-internal/engine/src/error/convert.rs index 0594c3d1d..d41ee117c 100644 --- a/crates/runvil-internal/engine/src/error/convert.rs +++ b/crates/runvil-internal/engine/src/error/convert.rs @@ -20,7 +20,7 @@ impl Error { /// Stamps the prompt's frontmatter name onto a parse failure raised by /// `run` itself: the prompt is already parsed at that point, so the - /// name is known and the location can name it. Any other variant passes + /// name is known and the failure's message can name it. Any other variant passes /// through unchanged. pub(crate) fn with_prompt_name(self, name: &str) -> Error { match self { @@ -32,8 +32,8 @@ impl Error { } /// Builds an internal-invariant failure, capturing the Rust source - /// position of the call site so [`crate::RunError::location`] can point - /// at the broken invariant. The message is owned when it names a value. + /// position of the call site so the run's failure message can point at + /// the broken invariant. The message is owned when it names a value. #[track_caller] pub(crate) fn internal(message: impl Into>) -> Error { let location = std::panic::Location::caller(); @@ -296,7 +296,8 @@ impl Error { /// `task_consumed` table rebuilds its variant from its `task` field, and /// every `cancelled` table maps to [`Error::Interrupted`]. A kind whose /// variant needs structure the table does not hold (the tool-scope - /// errors, `tasks_live`, `model_failure`, `model_required`, `internal`), + /// errors, `model_only_tool`, `tasks_live`, `model_failure`, + /// `model_required`, `internal`), /// or a `context_exhausted` or task table missing its field, keeps its /// message as a Lua failure. fn from_raised(raised: runvil_lua::Raised) -> Error { @@ -343,6 +344,7 @@ impl Error { }, runvil_lua::ErrorKind::OutOfScopeTool | runvil_lua::ErrorKind::UnboundTool + | runvil_lua::ErrorKind::ModelOnlyTool | runvil_lua::ErrorKind::TasksLive | runvil_lua::ErrorKind::Lua | runvil_lua::ErrorKind::ModelFailure diff --git a/crates/runvil-internal/engine/src/error/tests-chain.rs b/crates/runvil-internal/engine/src/error/tests-chain.rs index a46fac733..2be53be4a 100644 --- a/crates/runvil-internal/engine/src/error/tests-chain.rs +++ b/crates/runvil-internal/engine/src/error/tests-chain.rs @@ -1,8 +1,9 @@ //! The error-chain convention: a cause-bearing failure's message holds the //! cause's text, its source chain shows that text once, and its cause stays //! reachable through `originating_error()`; a variant that still keeps a `source()` -//! shows each segment of its chain once; an internal failure built with a -//! cause appends it once; and the store probe names a missing store. +//! shows each segment of its chain once, and so does the message a run +//! ends with; an internal failure built with a cause appends it once; and +//! the store probe names a missing store. use runvil_lua::VfsOp; use runvil_types::tools::{ToolError, ToolErrorKind}; @@ -114,18 +115,17 @@ fn no_segment_of_an_error_chain_repeats() { "required no source, since Display already holds the cause, actual a source on \ {error:?}" ); - let run_error = crate::RunError::from(error); - assert!( - run_error.originating_error().is_some(), - "the public wrapper reaches the cause: {run_error}" - ); - assert_eq!(assert_no_segment_repeats(&run_error), 1); + assert_eq!(assert_no_segment_repeats(&error), 1); } for error in source_bearing() { - let run_error = crate::RunError::from(error); assert!( - assert_no_segment_repeats(&run_error) > 1, - "required a chain of at least two segments to walk, actual one for {run_error:?}" + assert_no_segment_repeats(&error) > 1, + "required a chain of at least two segments to walk, actual one for {error:?}" + ); + assert_eq!( + error.message(), + chain(&error).join(": "), + "the run's message holds each segment of the chain once" ); } } @@ -145,15 +145,14 @@ fn an_internal_failure_built_with_a_cause_shows_its_text_once_and_keeps_it() { .as_deref(), Some("the cause") ); - let location = crate::RunError::from(error) - .location() - .expect("an internal fault reports a location"); assert_eq!( - location.path, - file!(), - "the path is the construction site's" + error.message(), + format!( + "internal invariant violated: a test invariant: the cause (at {}:{expected_line})", + file!() + ), + "the message ends with the construction site's file and line" ); - assert_eq!(location.line, Some(expected_line)); let bare = Error::internal("a test invariant"); assert!(bare.originating_error().is_none()); diff --git a/crates/runvil-internal/engine/src/error/tests.rs b/crates/runvil-internal/engine/src/error/tests.rs index f6a8ee552..6cdbe787f 100644 --- a/crates/runvil-internal/engine/src/error/tests.rs +++ b/crates/runvil-internal/engine/src/error/tests.rs @@ -1,6 +1,6 @@ //! Tests for the internal error: its source chains through the public -//! wrappers, its bridges from other crates' errors, and the locations and -//! classifications it reports through `RunError`. +//! wrappers, its bridges from other crates' errors, and the kinds and +//! messages, locations included, it ends a run with. use runvil_lua::Error as LuaError; @@ -35,20 +35,15 @@ fn context_exhaustion_maps_from_lua_and_classifies() { "a variant with no originating error has no cause" ); assert_eq!(error.to_string(), message); - let run_error = crate::RunError::from(error); - assert_eq!(run_error.kind(), crate::RunErrorKind::ContextExhausted); - assert!( - !run_error.is_retryable(), - "retrying an over-window request cannot succeed" - ); + assert_eq!(error.kind(), crate::RunErrorKind::ContextExhausted); + assert_eq!(error.message(), message); } } #[test] fn a_schema_build_failure_renders_its_refusal_and_keeps_it_as_the_cause() { - // The binding failure renders the schema refusal in its message and - // keeps the typed refusal as its cause, which the public `RunError` - // wrapper reaches. + // The binding failure renders the schema refusal in its message once + // and keeps the typed refusal as its cause. let schema_error = runvil_model_client::detail::tool_schema_new("echo", "", serde_json::json!([])) .expect_err("an array is not an object schema"); @@ -61,8 +56,8 @@ fn a_schema_build_failure_renders_its_refusal_and_keeps_it_as_the_cause() { "model-facing schema build failure for tool alias \"echo\": tool \"echo\" parameters \ schema must be a JSON object" ); - let run_error = crate::RunError::from(bind); - let cause = run_error + assert_eq!(bind.message(), bind.to_string()); + let cause = bind .originating_error() .and_then(|cause| cause.downcast_ref::()); assert!(cause.is_some(), "the schema refusal is the cause"); @@ -71,8 +66,7 @@ fn a_schema_build_failure_renders_its_refusal_and_keeps_it_as_the_cause() { #[test] fn lua_compile_keeps_the_originating_compiler_error_as_its_cause() { // A compile failure keeps the concrete `mlua` error as its cause - // instead of flattening it into `message` alone, and the public - // `RunError` wrapper reaches it. + // instead of flattening it into `message` alone. let compile = Error::LuaCompile { location: "section `S` prologue".to_owned(), source_line: 7, @@ -88,7 +82,6 @@ fn lua_compile_keeps_the_originating_compiler_error_as_its_cause() { .and_then(|cause| cause.downcast_ref::()) .expect("the compiler error is the cause"); assert!(matches!(cause, mlua::Error::SyntaxError { .. })); - assert!(crate::RunError::from(compile).originating_error().is_some()); } #[test] @@ -150,11 +143,10 @@ fn completion_errors_preserve_their_causes_across_the_error_type_bridge() { } #[test] -fn frontmatter_locations_surface_through_the_run_error() { +fn a_frontmatter_failure_message_ends_with_its_line_and_column() { // The parser's surfaced YAML position crosses the error-type - // bridge and lands on `RunError::location` for navigation. A - // frontmatter failure predates the prompt's name, so the path is - // the placeholder a caller replaces with its own label for the source. + // bridge and ends the run's failure message. A frontmatter failure + // predates the prompt's name, so the message names no prompt. let source = concat!( "---\n", "name: x\n", @@ -167,67 +159,57 @@ fn frontmatter_locations_surface_through_the_run_error() { let parse = Prompt::parse(source, "test") .0 .expect_err("a Pack id with spaces must be rejected"); - let run_error = crate::RunError::from(Error::from(parse)); - assert_eq!(run_error.kind(), crate::RunErrorKind::Parse); - let location = run_error - .location() - .expect("a parse failure reports a location"); - assert_eq!(location.line, Some(5)); - assert_eq!(location.column, Some(5)); - assert_eq!(location.span, None); + let error = Error::from(parse); + assert_eq!(error.kind(), crate::RunErrorKind::Parse); + assert_eq!(error.message(), format!("{error} (line 5, column 5)")); } #[test] -fn structured_locations_include_the_prompt_name_through_the_run_error() { - // A post-frontmatter parse failure reports the prompt's - // frontmatter name as the location's path, plus the offending - // span's line and column. +fn a_structured_failure_message_ends_with_the_prompt_name_and_position() { + // A post-frontmatter parse failure ends its message with the + // prompt's frontmatter name and the offending span's line and + // column. let source = "---\nname: dup\ndescription: d\n---\n\n# T\n\n## S\n\np\n\n## S\n\nq\n"; let parse = Prompt::parse(source, "test") .0 .expect_err("duplicate sibling sections must be rejected"); - let run_error = crate::RunError::from(Error::from(parse)); - let location = run_error - .location() - .expect("a structured parse failure reports a location"); - assert_eq!(location.path, "dup"); - assert_eq!(location.line, Some(12)); - assert_eq!(location.column, Some(1)); - assert!(location.span.is_some()); + let error = Error::from(parse); + assert_eq!( + error.message(), + format!("{error} (prompt dup, line 12, column 1)") + ); } #[test] -fn internal_faults_report_the_rust_file_and_line() { +fn an_internal_fault_message_ends_with_the_rust_file_and_line() { // An internal invariant failure locates itself in the Rust // source, captured at the construction site. let expected_line = line!() + 1; - let run_error = crate::RunError::from(Error::internal("a test invariant")); - let location = run_error - .location() - .expect("an internal fault reports a location"); - assert_eq!(location.path, file!(), "the path is the Rust source file"); - assert_eq!(location.line, Some(expected_line)); - assert_eq!(location.column, None); + let error = Error::internal("a test invariant"); + assert_eq!( + error.message(), + format!( + "internal invariant violated: a test invariant (at {}:{expected_line})", + file!() + ) + ); } #[test] -fn errors_without_a_location_return_none() { +fn a_failure_without_a_location_ends_its_message_with_its_own_text() { // Cancellation and the other non-positional kinds have no source - // position to navigate to. - let run_error = crate::RunError::from(Error::Interrupted); - assert!(run_error.location().is_none()); + // position to point at. + let error = Error::Interrupted; + assert_eq!(error.message(), error.to_string()); } #[test] fn interrupted_and_substitution_failures_classify_under_kinds_a_finished_run_reports() { - // A finished run reports cancellation as `RunResult::Cancelled`, so an - // interruption that reaches a `RunError` is the Engine's fault, and a - // failed `{{ }}` substitution is the author's program failing, as any + // A finished run reports cancellation as `RunOutcome::Cancelled`, so an + // interruption that reaches a failed outcome is the Engine's fault, and + // a failed `{{ }}` substitution is the author's program failing, as any // Lua fault is. - assert_eq!( - crate::RunError::from(Error::Interrupted).kind(), - crate::RunErrorKind::Internal - ); + assert_eq!(Error::Interrupted.kind(), crate::RunErrorKind::Internal); let (var, sys) = (serde_json::json!({}), serde_json::json!({})); let unset = |_: &str| -> Result> { Ok(None) }; let sources = crate::subst::Sources { @@ -241,26 +223,20 @@ fn interrupted_and_substitution_failures_classify_under_kinds_a_finished_run_rep let error = crate::subst::substitute("{{ var.missing }}", &sources) .expect_err("a missing key fails the substitution"); assert!(matches!(error, Error::Substitution(_)), "{error:?}"); - assert_eq!( - crate::RunError::from(error).kind(), - crate::RunErrorKind::Lua - ); + assert_eq!(error.kind(), crate::RunErrorKind::Lua); } #[test] fn requirements_unmet_classifies_and_reports_the_notice_as_its_message() { - // The refusal notice is the whole Display, and the kind classifies - // it for code. Retrying cannot help: the - // environment, not the transport, is what falls short. + // The refusal notice is the whole message, and the kind classifies + // it for code. let error = Error::RequirementsUnmet { - notice: "the environment cannot satisfy this prompt:\n- role 'analyst': requires a context of at least 200000 tokens; the current model provides 32000".to_owned(), + notice: "the environment cannot satisfy this prompt:\n- role 'strong': requires a context of at least 200000 tokens; the model filling it provides 32000".to_owned(), cause: None, }; - let run_error = crate::RunError::from(error); - assert_eq!(run_error.kind(), crate::RunErrorKind::RequirementsUnmet); - assert!(!run_error.is_retryable()); - assert!(run_error.location().is_none()); - assert!(run_error.to_string().contains("analyst")); + assert_eq!(error.kind(), crate::RunErrorKind::RequirementsUnmet); + assert_eq!(error.message(), error.to_string()); + assert!(error.message().contains("strong")); } #[test] @@ -277,11 +253,9 @@ fn the_call_depth_cap_is_a_quota_that_lua_still_reads_as_lua() { runvil_lua::ErrorValue::kind(&error), runvil_lua::ErrorKind::Lua ); - let run_error = crate::RunError::from(error); - assert_eq!(run_error.kind(), crate::RunErrorKind::Quota); - assert!(!run_error.is_retryable()); - assert!(run_error.location().is_none()); - assert!(run_error.originating_error().is_none()); + assert_eq!(error.kind(), crate::RunErrorKind::Quota); + assert_eq!(error.message(), text); + assert!(error.originating_error().is_none()); } } @@ -304,44 +278,15 @@ fn an_owned_lua_internal_message_reaches_the_engine_error_whole() { error.to_string(), format!("internal invariant violated: {message}") ); - let run_error = crate::RunError::from(error); - assert_eq!(run_error.kind(), crate::RunErrorKind::Internal); - let location = run_error - .location() - .expect("an internal fault reports a location"); + assert_eq!(error.kind(), crate::RunErrorKind::Internal); assert_eq!( - location.path, - file!(), - "the path is the construction site's file" - ); - assert_eq!(location.line, Some(expected_line)); -} - -#[test] -fn a_poisoned_model_set_maps_to_an_internal_error_and_never_a_completion_error() { - // The run's own model-set mutex failing is not a model failure and no - // prompt author can cause it: it maps to an internal error, and a - // caller never sees it as a retryable completion kind. - use crate::model::{ModelSet, ModelView}; - use std::sync::{Arc, Mutex}; - - let set = Arc::new(Mutex::new(ModelSet::default())); - let poisoner = Arc::clone(&set); - let _ = std::thread::spawn(move || { - let _guard = poisoner.lock(); - panic!("poison the model set"); - }) - .join(); - - let failure = set - .binding("writer") - .expect_err("a poisoned set cannot be read"); - let error = Error::from(runvil_lua::Error::from(failure)); - assert!( - matches!(&error, Error::Internal { message, .. } if message == "model set mutex was poisoned"), - "the lock failure is an internal error: {error:?}" + error.message(), + format!( + "internal invariant violated: {message} (at {}:{expected_line})", + file!() + ), + "the message ends with the construction site's file and line" ); - assert!(!crate::RunError::from(error).is_retryable()); } /// One failure of each store variant, with fields a table can carry. diff --git a/crates/runvil-internal/engine/src/error/value.rs b/crates/runvil-internal/engine/src/error/value.rs index 7617a35bc..1c6b0a5ba 100644 --- a/crates/runvil-internal/engine/src/error/value.rs +++ b/crates/runvil-internal/engine/src/error/value.rs @@ -34,6 +34,7 @@ impl runvil_lua::ErrorValue for Error { Error::TaskConsumed { .. } => ErrorKind::TaskConsumed, Error::OutOfScopeToolCall { .. } => ErrorKind::OutOfScopeTool, Error::UnboundToolCall { .. } => ErrorKind::UnboundTool, + Error::ModelOnlyToolCall { .. } => ErrorKind::ModelOnlyTool, Error::Tool { .. } => ErrorKind::Tool, Error::Store { .. } => ErrorKind::Store, Error::Parse(_) @@ -61,7 +62,9 @@ impl runvil_lua::ErrorValue for Error { )] }) .unwrap_or_default(), - Error::OutOfScopeToolCall { name, .. } | Error::UnboundToolCall { name, .. } => { + Error::OutOfScopeToolCall { name, .. } + | Error::UnboundToolCall { name, .. } + | Error::ModelOnlyToolCall { name, .. } => { vec![("name".to_owned(), ErrorField::String(name.clone()))] } Error::TasksLive { tasks } => { diff --git a/crates/runvil-internal/engine/src/execute.rs b/crates/runvil-internal/engine/src/execute.rs index 5e23ee997..1158fcd50 100644 --- a/crates/runvil-internal/engine/src/execute.rs +++ b/crates/runvil-internal/engine/src/execute.rs @@ -6,26 +6,29 @@ //! //! # Module layout //! -//! The run's outcome type ([`RunResult`]) is defined here; the rest lives in +//! The run's outcome type ([`RunOutcome`]) is defined here; the rest lives in //! focused private children: //! -//! - `bindings` - the run's model bindings, [`ModelBindings`]. -//! - `config` - the public [`RunContext`], whose `prepare` binds model -//! roles, and [`RunLimits`]; installing Packs and taking each run's -//! catalog from them is the Harness's job. -//! - `context` - the ambient `RunState` run state. -//! - `error` - the public [`RunError`] and its stable [`RunErrorKind`]. -//! - `fill` - prepare's model-role fill function. +//! - `config` - the public [`RunConfig`], which carries the caller's +//! role map for the five model roles, and [`RunLimits`]; installing +//! Packs and taking each run's catalog from them is the Harness's job. +//! - `context` - the ambient run state: the run-wide `SharedState` and +//! each chain's `ChainState`. +//! - `error` - the stable [`RunErrorKind`] of a failed outcome, and the +//! kind and message the Engine's own error ends a run with. +//! - `fill` - the fill function, which builds the model set from the +//! caller's role map when the run is built. //! - `infer` - the nested-inference round's answer. -//! - `requirements` - the preflight [`Requirements`] report. +//! - `requirements` - what a run's config lacks for its prompt, which the +//! run's first step reports as its refusal notice. //! - `run` - the boundary with the caller: the `Run` state machine with its -//! `step`, `resume`, and `cancel`, the `Step` it returns, and the effect -//! vocabulary (the `Effect` a leaf arm issues, its serializable -//! `EffectRecord`, and the `EffectAnswer` the caller returns). +//! `step` and `resume`, the `Step` it returns, and the effect +//! vocabulary (the `Effect` a leaf arm issues, its JSON record, and the +//! `EffectAnswer` the caller returns). //! - `scheduler` - the chain scheduler driving the coroutine protocol: //! the H1 pass, the walk, call chains, fanout, and the `chat` and -//! `tool_call` rounds that `models.loop` and a model handle's `loop` -//! yield over the section's `messages`. +//! `tool_call` rounds that `models.loop` +//! yields over the section's `messages`. //! - `scope` - the merge that builds and checks a section entry's frozen //! sets, and the schemas a round projects from its Static Set. //! - `section_context` - the per-section `SectionContext` frame the @@ -37,7 +40,6 @@ //! completion goes through. //! - `walk_target` - the walk-target resolution helpers. -mod bindings; mod config; pub(crate) mod context; mod error; @@ -53,15 +55,13 @@ mod turn; mod walk_target; // Public API surface. -pub use bindings::ModelBindings; -pub use config::{RunContext, RunLimits}; -pub use error::{RunError, RunErrorKind, SourceLocation}; -pub use requirements::{ - MissingService, RequirementCheck, Requirements, UnavailablePack, UnmetRequirement, UsablePack, -}; +pub use config::{RunConfig, RunLimits}; +pub use error::RunErrorKind; +#[cfg(test)] +pub(crate) use run::{AnswerRecord, ChatAnswerRecord, EffectRecord}; pub use run::{ - AnswerRecord, ChatAnswerRecord, Effect, EffectAnswer, EffectId, EffectRecord, FailureRecord, - IssuedEffect, PackToolSets, Round, Run, Step, ToolAnswerRecord, ToolCallOrigin, ToolCaller, + Effect, EffectAnswer, EffectId, IssuedEffect, PackToolSets, Round, Run, Step, ToolCallOrigin, + ToolCaller, }; // The store vocabulary a `Vfs` effect holds and its answer returns, for // the Engine's own store handling; other crates name it from `runvil_lua`. @@ -97,31 +97,45 @@ pub fn perform_vfs_op( /// Every outcome is a value of this enum. That includes a prompt that /// declines its task: a decline is not an error and not a cancellation, so /// the run still completes, with the prompt's reason for declining as the -/// [`Completed`](RunResult::Completed) text. The variant tells code what +/// [`Completed`](RunOutcome::Completed) text. The variant tells code what /// happened, and the payload explains it to people and models. /// -/// This enum is the one outcome type that is `#[non_exhaustive]`, so a -/// `match` on it needs a wildcard arm, while [`Step`], [`Effect`], and -/// [`EffectAnswer`] are deliberately closed. Those three are the caller's -/// protocol: its loop names every kind, so a new kind there must break its -/// build rather than reach code that cannot handle it. A run's outcome is -/// the exception because the caller consumes it as a result, and a wildcard -/// arm that treats an unfamiliar outcome as a failure is a sound default -/// where an unfamiliar effect kind is not. -/// -/// A failure's [`RunError`] has a [`kind`](RunError::kind) that classifies -/// the failure by condition; [`RunErrorKind`] documents each kind. -#[derive(Debug)] -#[non_exhaustive] -pub enum RunResult { - /// The run completed with its final text. - Completed(String), +/// Closed on purpose, like [`Step`]: a new outcome breaks every caller's +/// build rather than reach code that cannot handle it. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum RunOutcome { + /// The run completed. + Completed { + /// The run's final text. + final_text: String, + }, + /// The run failed. + Failed { + /// The failure's stable classification; [`RunErrorKind`] documents + /// each kind. + kind: RunErrorKind, + /// The failure's self-contained message, with the text of each + /// cause it keeps and, for a parse or internal failure, where it + /// happened. + message: String, + }, /// The caller cancelled the run or dropped an effect the run was waiting /// on and the cancelled error went uncaught, or a block raised a caught /// `cancelled` error value again after another suspending call. Cancelled, - /// The run failed. The error's kind classifies the failure. - Failure(RunError), +} + +impl RunOutcome { + /// The text that names this outcome: `completed`, `failed`, or + /// `cancelled`. + #[must_use] + pub const fn as_str(&self) -> &'static str { + match self { + Self::Completed { .. } => "completed", + Self::Failed { .. } => "failed", + Self::Cancelled => "cancelled", + } + } } #[cfg(test)] diff --git a/crates/runvil-internal/engine/src/execute/bindings.rs b/crates/runvil-internal/engine/src/execute/bindings.rs deleted file mode 100644 index 26a8b9098..000000000 --- a/crates/runvil-internal/engine/src/execute/bindings.rs +++ /dev/null @@ -1,114 +0,0 @@ -//! The run's model bindings: [`ModelBindings`]. - -use std::collections::BTreeMap; - -use crate::model::{ModelId, ModelInfo}; - -/// The models bound for a run: which model each declared role uses, and -/// the model info of every model a role is bound to. -/// -/// [`prepare`](super::RunContext::prepare) fills these bindings. It binds -/// every declared role to the current model the caller passes it. -/// -/// A lookup goes from a role label to a model identity, and from that -/// identity to the model's info. -#[derive(Debug, Clone, Default, PartialEq, Eq)] -pub struct ModelBindings { - /// The decision: role label to the bound model's identity. - roles: BTreeMap, - /// The info of every model a role is bound to: identity to model info. - models: BTreeMap, -} - -impl ModelBindings { - /// Binds the role `label` to `model`, recording the model info under - /// its identity. - pub(super) fn bind(&mut self, label: &str, model: ModelInfo) { - self.roles.insert(label.to_owned(), model.id().clone()); - self.models.entry(model.id().clone()).or_insert(model); - } - - /// Returns the identity of the model bound to the role `label`, or - /// `None` when the role was not filled. - #[must_use] - pub fn role_id(&self, label: &str) -> Option<&ModelId> { - self.roles.get(label) - } - - /// Returns the info of the model bound to the role `label`, or - /// `None` when the role was not filled. - #[must_use] - pub fn resolve(&self, label: &str) -> Option<&ModelInfo> { - self.roles.get(label).and_then(|id| self.models.get(id)) - } - - /// Returns the info of the model with the identity `id`, or `None` - /// when no role is bound to that model. - /// - /// `None` does not mean the run may not use the model: a catalog model - /// that a UI snapshot reaches by raw id is not in this map either, - /// because only a role binding puts one here. - #[must_use] - pub fn model(&self, id: &ModelId) -> Option<&ModelInfo> { - self.models.get(id) - } - - /// Returns the number of bound roles. - #[must_use] - pub fn len(&self) -> usize { - self.roles.len() - } - - /// Returns whether the set of bound roles is empty. - #[must_use] - pub fn is_empty(&self) -> bool { - self.roles.is_empty() - } -} - -#[cfg(test)] -mod tests { - use std::num::NonZeroU32; - - use super::*; - use crate::model::ThinkingMode; - - fn model_info(name: &str) -> ModelInfo { - ModelInfo::new( - ModelId::gateway(name).expect("the test id is valid"), - "A test model", - NonZeroU32::new(32_000).expect("the context window is non-zero"), - ThinkingMode::Switchable, - ) - } - - #[test] - fn an_empty_binding_set_resolves_nothing() { - let bindings = ModelBindings::default(); - assert!(bindings.is_empty()); - assert_eq!(bindings.len(), 0); - assert!(bindings.role_id("analyst").is_none()); - assert!(bindings.resolve("analyst").is_none()); - } - - #[test] - fn two_roles_bound_to_one_model_share_one_model_info_entry() { - // The trivial fill binds every role to the same model, and the - // model info table holds it once. - let model = model_info("current"); - let mut bindings = ModelBindings::default(); - bindings.bind("analyst", model.clone()); - bindings.bind("triage", model.clone()); - assert_eq!(bindings.len(), 2); - assert_eq!(bindings.role_id("analyst"), Some(model.id())); - assert_eq!(bindings.role_id("triage"), Some(model.id())); - assert_eq!(bindings.resolve("analyst"), Some(&model)); - assert_eq!(bindings.resolve("triage"), Some(&model)); - assert_eq!(bindings.model(model.id()), Some(&model)); - assert!( - bindings - .model(&ModelId::gateway("other").expect("valid")) - .is_none() - ); - } -} diff --git a/crates/runvil-internal/engine/src/execute/config.rs b/crates/runvil-internal/engine/src/execute/config.rs index 6bb2d2e2e..39681b8ba 100644 --- a/crates/runvil-internal/engine/src/execute/config.rs +++ b/crates/runvil-internal/engine/src/execute/config.rs @@ -1,4 +1,4 @@ -//! Per-run context and resource limits: [`RunContext`] and [`RunLimits`]. +//! Per-run config and resource limits: [`RunConfig`] and [`RunLimits`]. use std::collections::BTreeMap; use std::fmt; @@ -7,10 +7,8 @@ use std::fmt; mod limits; use runvil_types::cancel::CancelHandle; -use runvil_types::emitter::DebugMode; use runvil_types::packs::{PackId, Prelude}; -use runvil_types::replay::Flags; -use runvil_types::seed::{RunSeed, RunUid}; +use runvil_types::seed::RunSeed; use runvil_types::timestamp::Timestamp; use runvil_types::tools::ToolCatalog; use runvil_vfs::VfsRef; @@ -18,47 +16,40 @@ use runvil_vfs::VfsRef; pub use limits::RunLimits; use crate::lua::RosterEntry; -use crate::model::ModelInfo; -use crate::parser::Prompt; - -use super::bindings::ModelBindings; -use super::fill; -use super::requirements::{Requirements, UnavailablePack, UsablePack}; +use crate::model::{ModelInfo, ModelRole}; /// The inputs for one run of a prompt. /// -/// The caller builds a context with `new` and the `with_*` builder +/// The caller builds a config with `new` and the `with_*` builder /// methods, among them the run's tool catalog -/// ([`with_tools`](RunContext::with_tools)), Pack roster -/// ([`with_packs`](RunContext::with_packs)), and Pack preludes -/// ([`with_preludes`](RunContext::with_preludes)). Then it calls -/// [`prepare`](RunContext::prepare), which binds the prompt's model roles -/// to the current model and reports what the caller must still satisfy. The -/// Engine owns the context for the run. Each context belongs to exactly one -/// run. +/// ([`with_tools`](RunConfig::with_tools)), Pack roster +/// ([`with_packs`](RunConfig::with_packs)), Pack preludes +/// ([`with_preludes`](RunConfig::with_preludes)), and role map +/// ([`with_models`](RunConfig::with_models)). Building the run fills the +/// five model roles from the role map, and the run's first step checks +/// the config against what the prompt requires. The Engine owns the +/// config for the run. Each config belongs to exactly one run. /// -/// The context holds only the Engine's input. The observer, client, tool +/// The config holds only the Engine's input. The observer, client, tool /// implementations, broker, and capture belong to the caller. The Engine /// reports events and issues effects as values, and returns each one to /// the caller from `step`. /// /// The caller supplies the run's clock and randomness through `new`. -pub struct RunContext { - /// The caller's run label, moved into `RunState::execution` at start and - /// read nowhere else. +pub struct RunConfig { + /// The caller's run label, moved into `SharedState::execution` at start + /// and read nowhere else. pub(super) name: String, - /// The run's seed, as the caller supplied it: the source of the run id, - /// the untrusted-envelope nonce, and every other in-run random choice. + /// The run's seed, as the caller supplied it: the source of the + /// untrusted-envelope nonce and every other in-run random choice. pub(super) seed: RunSeed, - /// The behavior flags the run records; empty by default. - flags: Flags, /// When the run began, as the caller stamped it: rendered as `sys.when` /// in every section, the H1 pass included. pub(super) started_at: Timestamp, /// Where the root task's provenance sequence starts; set by `with_provenance_start`. pub(super) provenance_start: u32, /// Whether the run reports raw model-round bodies as events; set by `with_report_debug`. - pub(super) report_debug: DebugMode, + pub(super) report_debug: bool, /// The run's cancel flag, minted by `new` and replaced by `with_cancel`. pub(super) cancel: CancelHandle, /// The run's resource limits; set by `with_limits`. @@ -67,47 +58,47 @@ pub struct RunContext { pub(super) ui: Option, /// The run's whole filesystem; set by `with_vfs`. pub(super) vfs: VfsRef, - /// The run's model bindings; filled by [`prepare`](RunContext::prepare). - pub(super) model_bindings: ModelBindings, + /// The caller's role map, which fills the run's five roles; set by + /// `with_models`. + pub(super) models: BTreeMap, /// The run's tool catalog; set by `with_tools`. pub(super) tools: ToolCatalog, /// Each roster Pack by name, with its vendor when it can serve and the - /// reason when it can't; set by `with_packs`. - pub(super) packs: BTreeMap, + /// reason when it can't; set by `with_packs`, and `None` without a + /// roster. + pub(super) packs: Option>, /// The run's Pack preludes, in declaration order; set by `with_preludes`. pub(super) preludes: Vec, } -impl RunContext { - /// Builds a context for a run named `name`, with the given `seed` and +impl RunConfig { + /// Builds a config for a run named `name`, with the given `seed` and /// `started_at`. /// - /// The new context has a fresh cancel flag, a `ui` snapshot of `None`, - /// debug reporting set to `DebugMode::Off`, default [`RunLimits`], empty - /// [`Flags`], and the default filesystem, which is a fresh memory store - /// at `/`. + /// The new config has a fresh cancel flag, a `ui` snapshot of `None`, + /// debug reporting off, default [`RunLimits`], and the default + /// filesystem, which is a fresh memory store at `/`. /// - /// `seed` is the source of the run id and the untrusted-envelope nonce. - /// For a live run, the caller must draw its 32 bytes from a CSPRNG, + /// `seed` is the source of the untrusted-envelope nonce. For a live + /// run, the caller must draw its 32 bytes from a CSPRNG, /// because a predictable seed makes a guessable nonce. `started_at` is /// the instant every section reads as `sys.when`. The caller records /// both values, and a replay passes them back verbatim. #[must_use] - pub fn new(name: impl Into, seed: RunSeed, started_at: Timestamp) -> RunContext { - RunContext { + pub fn new(name: impl Into, seed: RunSeed, started_at: Timestamp) -> RunConfig { + RunConfig { name: name.into(), seed, - flags: Flags::EMPTY, started_at, provenance_start: 0, - report_debug: DebugMode::Off, + report_debug: false, cancel: CancelHandle::new(), limits: RunLimits::default(), ui: None, vfs: VfsRef::default(), - model_bindings: ModelBindings::default(), + models: BTreeMap::new(), tools: ToolCatalog::default(), - packs: BTreeMap::new(), + packs: None, preludes: Vec::new(), } } @@ -115,30 +106,41 @@ impl RunContext { /// Sets whether the run reports each model round's raw request and /// response bodies as `Request` and `Response` events. /// - /// The default, [`DebugMode::Off`], turns both events off. A caller that - /// wants the pair in the event stream, such as for a debug capture, - /// passes [`DebugMode::On`]. + /// The default, `false`, turns both events off, so the rounds never + /// clone a body for them. A caller that wants the pair in the event + /// stream, such as for a debug capture, passes `true`. #[must_use] - pub fn with_report_debug(mut self, mode: DebugMode) -> RunContext { - self.report_debug = mode; + pub fn with_report_debug(mut self, report: bool) -> RunConfig { + self.report_debug = report; self } /// Sets the run's cancel flag, replacing the one that `new` created. /// + /// The caller keeps its own clone of `handle` and cancels the run by + /// setting it, from any thread. The flag `new` created is the run's + /// alone, so a run built without this call cannot be cancelled. + /// /// The flag is a synchronous [`CancelHandle`] that the Engine polls - /// between chain steps and from the Lua instruction hook. A caller that - /// cancels through an awaitable token must bridge the token to this - /// flag by setting the flag when the token fires. + /// between chain steps and from the Lua instruction hook, so running + /// Lua code sees it too. The next [`step`](super::Run::step) tears down + /// every chain and lists every effect still outstanding in its + /// `withdrawn`. The run then reports itself as cancelled once every + /// outstanding effect is answered. Cancellation is a request that later + /// steps complete: the caller answers each effect it abandons with + /// [`EffectAnswer::Dropped`](super::EffectAnswer::Dropped). + /// + /// A caller that cancels through an awaitable token must bridge the + /// token to this flag by setting the flag when the token fires. #[must_use] - pub fn with_cancel(mut self, handle: CancelHandle) -> RunContext { + pub fn with_cancel(mut self, handle: CancelHandle) -> RunConfig { self.cancel = handle; self } /// Sets the resource limits honored across the run. #[must_use] - pub fn with_limits(mut self, limits: RunLimits) -> RunContext { + pub fn with_limits(mut self, limits: RunLimits) -> RunConfig { self.limits = limits; self } @@ -149,27 +151,17 @@ impl RunContext { /// With a snapshot set, every section VM gets a `ui()` global that /// returns it. The caller takes the snapshot at run start, so a change /// to application state takes effect on the next run. A snapshot also - /// makes `models.get` resolve an alias outside the prompt's declarations - /// as a raw model id. So a prompt can run - /// `models.get(ui().selected_model):loop()` and skip - /// declaring its model. The default is `None`, which leaves the `ui` - /// global absent and limits `models.get` to declared aliases. + /// makes `models.get` resolve a label outside the five roles as a raw + /// gateway model id. So a prompt can run + /// `models.loop(models.get(ui().selected_model))` and run on the model the + /// user picked. The default is `None`, which leaves the `ui` global + /// absent and limits `models.get` to the five roles. #[must_use] - pub fn with_ui(mut self, snapshot: serde_json::Value) -> RunContext { + pub fn with_ui(mut self, snapshot: serde_json::Value) -> RunConfig { self.ui = Some(snapshot); self } - /// Sets the behavior flags the run records. - /// - /// A live run keeps the default, the empty set. A replay passes back the - /// set the original run recorded. - #[must_use] - pub fn with_flags(mut self, flags: Flags) -> RunContext { - self.flags = flags; - self - } - /// Sets where the root task's provenance sequence starts. /// /// The default, 0, fits a run recorded on its own. When the caller @@ -179,7 +171,7 @@ impl RunContext { /// past them and every `(task, seq)` in the stream stays unique. /// Spawned tasks count from zero. #[must_use] - pub fn with_provenance_start(mut self, start: u32) -> RunContext { + pub fn with_provenance_start(mut self, start: u32) -> RunConfig { self.provenance_start = start; self } @@ -192,10 +184,10 @@ impl RunContext { /// /// Every tool call effect carries an access to this same filesystem, so /// the tools and the run share one set of files. The caller seeds files - /// before the run and extracts output after it through the prepared - /// handle, returned by [`vfs`](RunContext::vfs). + /// before the run and extracts output after it through its own clone of + /// `vfs`. #[must_use] - pub fn with_vfs(mut self, vfs: VfsRef) -> RunContext { + pub fn with_vfs(mut self, vfs: VfsRef) -> RunConfig { self.vfs = vfs; self } @@ -204,23 +196,26 @@ impl RunContext { /// every Pack the caller can serve this run, declared by the prompt or /// not. The catalog lists every tool the run can use: Lua calls one by id, /// and a section offers one to its model with `tools.offer`. The caller - /// performs every call. Read it back with - /// [`tools`](RunContext::tools). + /// performs every call. #[must_use] - pub fn with_tools(mut self, tools: ToolCatalog) -> RunContext { + pub fn with_tools(mut self, tools: ToolCatalog) -> RunConfig { self.tools = tools; self } /// Sets the run's roster: every installed Pack that can serve the run, /// declared by the prompt or not, with its vendor, and every one that - /// can't, with the reason. + /// can't, with the reason. A Pack listed twice keeps its last entry. /// /// Each usable Pack gets an object in the run's Lua, with or without /// tools. An unavailable one gets none, and offering it raises its - /// reason. A name in both lists counts as unavailable. Without a roster, - /// the run's Lua sees the declared Packs and the Packs with a tool - /// in the catalog. + /// reason. Without a roster, the run's Lua sees the declared Packs and + /// the Packs with a tool in the catalog. + /// + /// The run's first step checks each Pack the prompt declares against + /// the roster: one the roster lacks is missing, and one it lists as + /// unavailable fails the run with its reason. A run given no roster + /// checks no declared Pack. /// /// The caller reports each usable Pack's vendor, as it picked the /// Pack. Every tool-call and tool-result event for a catalog tool of @@ -229,28 +224,9 @@ impl RunContext { #[must_use] pub fn with_packs( mut self, - usable: Vec, - unavailable: Vec, - ) -> RunContext { - self.packs = usable - .into_iter() - .map(|entry| { - ( - entry.pack, - RosterEntry::Usable { - vendor: entry.vendor, - }, - ) - }) - .chain(unavailable.into_iter().map(|entry| { - ( - entry.pack, - RosterEntry::Unavailable { - reason: entry.reason, - }, - ) - })) - .collect(); + roster: impl IntoIterator, + ) -> RunConfig { + self.packs = Some(roster.into_iter().collect()); self } @@ -271,114 +247,42 @@ impl RunContext { /// The failure happens when the first section VM is set up, before the /// run issues any effect. #[must_use] - pub fn with_preludes(mut self, preludes: Vec) -> RunContext { + pub fn with_preludes(mut self, preludes: Vec) -> RunConfig { self.preludes = preludes; self } - /// Binds `prompt`'s model roles to `model`, the run's current model, - /// which the caller chooses, and reports what the caller must still - /// satisfy, returning the context beside the report. + /// Sets the caller's role map, which fills the run's five model roles + /// (`default`, `strong`, `small`, `fast` and `creative`) when the run + /// is built. /// - /// `prepare` leaves the filesystem as given. The caller checks before - /// `prepare` that each Pack the prompt declares can serve, and merges - /// that report into the one `prepare` returns. + /// A role's model is its own entry in `models`, else the `default` + /// entry, so one entry for `default` fills all five roles. Every prompt + /// gets all five roles, whether or not it declares them. A role's + /// declared `thinking` requirement turns thinking on or off for its + /// rounds. /// - /// Every declared model role binds to `model`. `prepare` then checks - /// each role's hard keywords (`thinking`, `no-thinking`) and context - /// minimum against the model's info. Each failed check is listed in - /// [`Requirements::unmet_models`] with the required and actual values, - /// and the role stays bound to `model`. Soft keywords record the - /// author's intent only, so `prepare` skips them. When `model` is - /// `None`, `prepare` skips the fill and the checks, the model bindings - /// stay empty, and selecting a declared role at run time fails. + /// The run's first step checks each declared role's `min_context` and + /// `thinking` requirements against the info of the model that fills + /// it, and fails the run naming each one that falls short, with the + /// required and actual values. /// - /// `prepare` also lists in [`Requirements::reserved`] each reserved - /// Pack id, such as `adhoc`, that a tool of the catalog or a Pack of - /// the roster, usable or not, has. + /// An empty `models`, the default, means the caller has no model: the + /// roles stay empty, nothing is checked, and a model round at run time + /// fails. A non-empty `models` with no `default` entry leaves the roles + /// empty and fails the run's first step. #[must_use] - pub fn prepare( - mut self, - prompt: &Prompt, - model: Option, - ) -> (RunContext, Requirements) { - let mut requirements = Requirements::default(); - self.model_bindings = fill::fill_model_bindings(prompt, model, &mut requirements); - requirements.reserved = self - .tools - .tools() - .iter() - .map(|tool| tool.id.pack()) - .chain(self.packs.keys().cloned()) - .filter(PackId::is_reserved) - .collect(); - (self, requirements) - } - - /// Returns the run's VFS handle, set with [`with_vfs`](RunContext::with_vfs). - #[must_use] - pub fn vfs(&self) -> &VfsRef { - &self.vfs - } - - /// Returns the run's [`ModelBindings`], which - /// [`prepare`](RunContext::prepare) fills. - #[must_use] - pub fn model_bindings(&self) -> &ModelBindings { - &self.model_bindings - } - - /// Binds the role `label` to `model` as the prepare's fill would, for a - /// test that starts the run without the prepare. - #[cfg(test)] - pub(crate) fn bind_role_for_test(&mut self, label: &str, model: ModelInfo) { - self.model_bindings.bind(label, model); - } - - /// Returns the run's tool catalog, set with - /// [`with_tools`](RunContext::with_tools). It is empty by default. - #[must_use] - pub fn tools(&self) -> &ToolCatalog { - &self.tools - } - - /// Returns the run's seed, as the caller supplied it. - #[must_use] - pub fn seed(&self) -> RunSeed { - self.seed - } - - /// Returns the run's id, derived from its seed. - /// - /// The id is one-way, so it does not reveal the seed or the nonce. It - /// appears in a debug rendering only: events and reports carry the - /// run's `name`. A replay under the recorded seed has the same id. - #[must_use] - pub(crate) fn uid(&self) -> RunUid { - RunUid::from_seed(&self.seed) - } - - /// Returns the behavior flags the run records. Set them with - /// [`with_flags`](RunContext::with_flags). - #[must_use] - pub fn flags(&self) -> Flags { - self.flags - } - - /// Returns when the run began, as the caller stamped it. - #[must_use] - pub fn started_at(&self) -> Timestamp { - self.started_at + pub fn with_models(mut self, models: BTreeMap) -> RunConfig { + self.models = models; + self } } -impl fmt::Debug for RunContext { +impl fmt::Debug for RunConfig { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - let mut state = f.debug_struct("RunContext"); + let mut state = f.debug_struct("RunConfig"); state .field("name", &self.name) - .field("uid", &self.uid()) - .field("flags", &self.flags) .field("started_at", &self.started_at) .field("provenance_start", &self.provenance_start) .field("report_debug", &self.report_debug) @@ -386,7 +290,7 @@ impl fmt::Debug for RunContext { .field("limits", &self.limits) .field("ui", &self.ui) .field("vfs", &self.vfs) - .field("model_bindings", &self.model_bindings) + .field("models", &self.models) .field("tools", &self.tools) .field("packs", &self.packs) .field( diff --git a/crates/runvil-internal/engine/src/execute/context-bound.rs b/crates/runvil-internal/engine/src/execute/context-bound.rs index 6d1e31c22..761462d90 100644 --- a/crates/runvil-internal/engine/src/execute/context-bound.rs +++ b/crates/runvil-internal/engine/src/execute/context-bound.rs @@ -1,9 +1,9 @@ -//! The run-scoped tool and model sets and the `argv` derivation: the pieces -//! `RunState::new` assembles once per run. +//! The run-scoped tool set and the `argv` derivation: the pieces +//! `ChainState::root` assembles once per run. //! -//! The sets are built once. [`derive_argv`] is not: `RunState::new` calls it -//! for the run's own args, and every args fork calls it again, so a -//! contained chain parses the input it was given. +//! The tool set is built once. [`derive_argv`] is not: `ChainState::root` +//! calls it for the run's own args, and every args fork calls it again, so +//! a contained chain parses the input it was given. use std::collections::BTreeMap; @@ -11,13 +11,10 @@ use runvil_types::packs::PackId; use runvil_types::tools::ToolCatalog; use crate::lua::{RosterEntry, ToolSet}; -use crate::model::{ModelBinding, ModelInvocation, ModelSet}; -use crate::parser::{ModelKeyword, Prompt, detail}; - -use super::super::bindings::ModelBindings; +use crate::parser::Prompt; /// Builds the run's shared tool set: the frontmatter's declared Packs, -/// and the run catalog and roster, all moved out of the context. Nothing +/// and the run catalog and roster, all moved out of the config. Nothing /// is bound here: each section entry binds the tools it offers when it /// freezes its sets. pub(super) fn run_tool_set( @@ -37,48 +34,6 @@ pub(super) fn run_tool_set( ) } -/// Builds the run's shared model set from the prepared `bindings`: every -/// filled role becomes a binding under its label, holding the role's -/// keyword set (the handle's `capabilities`), the hard-keyword thinking -/// switch as the frozen invocation, the role's declared description (else -/// the model info's), the catalog model info's context window, the model -/// info's provider id when the catalog names one, and the model info itself, -/// which a section entry's render receives. Unfilled roles produce no binding: -/// `models.use` on the label fails at run time. -pub(super) fn bound_model_set(prompt: &Prompt, bindings: &ModelBindings) -> ModelSet { - let mut set = ModelSet::default(); - for (label, role) in prompt.frontmatter().models() { - let Some(info) = bindings.resolve(label) else { - continue; - }; - let mut thinking = None; - for keyword in role.keywords() { - match keyword { - ModelKeyword::Thinking => thinking = Some(true), - ModelKeyword::NoThinking => thinking = Some(false), - _ => {} - } - } - let binding = ModelBinding::new( - label, - role.description().unwrap_or_else(|| info.description()), - info.id().clone(), - ModelInvocation::new().with_thinking(thinking), - info.context(), - ) - .with_capabilities( - role.keywords() - .iter() - .map(|keyword| detail::keyword_name(*keyword)) - .map(str::to_owned) - .collect(), - ) - .with_catalog_info(info.clone()); - set.push(binding); - } - set -} - /// Derives a section's `argv` from the args string under the prompt's /// declaration: a default-declared prompt wraps the interface prose into /// the default shape (`argv.prose`, with the empty string present, not diff --git a/crates/runvil-internal/engine/src/execute/context-tests.rs b/crates/runvil-internal/engine/src/execute/context-tests.rs index c735b9be6..ef121ffca 100644 --- a/crates/runvil-internal/engine/src/execute/context-tests.rs +++ b/crates/runvil-internal/engine/src/execute/context-tests.rs @@ -1,6 +1,8 @@ -//! Tests for `RunContext` construction, forking, task sequence seeding, +//! Tests for `ChainState` construction, forking, task sequence seeding, //! and the model set a run binds from its filled roles. +use std::collections::BTreeMap; + use runvil_types::packs::PackId; use runvil_types::tools::ToolId; @@ -8,9 +10,8 @@ use serde_json::json; use super::*; use crate::execute::section_vm::VmSeed; -use crate::execute::{UnavailablePack, UsablePack}; use crate::lua::{Argv, RosterEntry}; -use crate::model::ThinkingMode; +use crate::model::{ModelRole, ThinkingMode}; use crate::test_support::fresh_access; use crate::test_support::scenario::{TEST_SEED, test_model_with}; @@ -24,10 +25,10 @@ fn test_prompt() -> Prompt { .expect("the test prompt parses") } -fn test_state(prompt: &Prompt) -> RunState { +fn test_state(prompt: &Prompt) -> ChainState { state_over( prompt, - RunContext::new( + RunConfig::new( "run-context-test", TEST_SEED, runvil_types::timestamp::Timestamp::UNIX_EPOCH, @@ -37,12 +38,12 @@ fn test_state(prompt: &Prompt) -> RunState { /// The state a run of `prompt` assembles from `config`, as `Run::new` /// assembles it. -fn state_over(prompt: &Prompt, config: RunContext) -> RunState { - RunState::new(Arc::new(prompt.clone()), "", config) +fn state_over(prompt: &Prompt, config: RunConfig) -> ChainState { + ChainState::root(Arc::new(prompt.clone()), "", config) } #[test] -fn new_builds_a_context_over_the_prompt() { +fn root_builds_a_context_over_the_prompt() { let prompt = test_prompt(); let ctx = test_state(&prompt); assert_eq!(ctx.prompt().title(), prompt.title()); @@ -59,7 +60,7 @@ fn accessor_returns_the_run_prompt() { fn clones_share_the_prompt_allocation() { let ctx = test_state(&test_prompt()); let clone = ctx.clone(); - assert!(Arc::ptr_eq(&ctx.prompt, &clone.prompt)); + assert!(Arc::ptr_eq(ctx.prompt(), clone.prompt())); } #[test] @@ -85,13 +86,14 @@ fn vm_setup_hands_the_vm_the_runs_argv_in_the_mode_the_caller_asks_for() { expected.is_some(), "the fixture's args must derive an argv, or neither mode is observable" ); - let frozen = ctx.vm_setup(&sys, VmSeed::default(), &access, "Only", false); + let selection = Arc::new(Mutex::new(None)); + let frozen = ctx.vm_setup(&sys, VmSeed::default(), &access, &selection, "Only", false); assert!( matches!(frozen.argv, Argv::Frozen(value) if value == expected), "an unprivileged constructor must get the run's argv frozen: {:?}", frozen.argv ); - let writable = ctx.vm_setup(&sys, VmSeed::default(), &access, "Only", true); + let writable = ctx.vm_setup(&sys, VmSeed::default(), &access, &selection, "Only", true); assert!( matches!(writable.argv, Argv::Writable(value) if value == expected), "the H1 pass's flag must get the run's argv writable: {:?}", @@ -104,23 +106,23 @@ fn forks_swap_only_their_own_fields() { let ctx = test_state(&test_prompt()); let chain = ctx.with_args("chain-args"); assert_eq!(&**chain.args(), "chain-args"); - assert!(Arc::ptr_eq(&ctx.prompt, &chain.prompt)); + assert!(Arc::ptr_eq(&ctx.shared, &chain.shared)); assert_eq!(&**ctx.args(), ""); let turns = Arc::new(AtomicU32::new(7)); let task: TaskId = "0.4".parse().expect("a task id parses"); let arm = ctx.with_task(task.clone(), Arc::clone(&turns)); assert!(Arc::ptr_eq(arm.turns(), &turns)); - assert!(Arc::ptr_eq(&ctx.prompt, &arm.prompt)); + assert!(Arc::ptr_eq(&ctx.shared, &arm.shared)); assert_eq!(arm.emitter().task(), &task); assert_eq!(ctx.emitter().task(), &TaskId::from(ChainId::root())); } #[test] -fn the_root_task_sequence_starts_where_the_context_says() { +fn the_root_task_sequence_starts_where_the_config_says() { let ctx = state_over( &test_prompt(), - RunContext::new( + RunConfig::new( "run-context-test", TEST_SEED, runvil_types::timestamp::Timestamp::UNIX_EPOCH, @@ -148,7 +150,7 @@ fn a_task_fork_reports_into_the_shared_buffer_under_its_own_task() { arm.emitter() .report("Only", runvil_types::event::lifecycle::SECTION_STARTED); // Both emitters share one buffer, drained through either context. - let events = arm.events.take(); + let events = arm.take_events(); assert_eq!(events.len(), 2); assert_eq!(events[0].provenance().task, TaskId::from(ChainId::root())); assert_eq!(events[1].provenance().task, task); @@ -157,52 +159,49 @@ fn a_task_fork_reports_into_the_shared_buffer_under_its_own_task() { #[test] fn a_filled_role_binds_with_the_catalog_info_its_render_receives() { - let source = concat!( - "---\nname: t\ndescription: d\nrunvil: 0\nmodels:\n writer: {}\n---\n\n", - "# Title\n\n## Only\n\ndone\n", - ); - let prompt = Prompt::parse(source, "run-context-test") - .0 - .expect("the test prompt parses"); + let prompt = test_prompt(); let info = test_model_with("catalog-model", 65_536, ThinkingMode::Switchable).with_provider("acme"); - let mut ctx = RunContext::new( + let ctx = RunConfig::new( "run-context-test", TEST_SEED, runvil_types::timestamp::Timestamp::UNIX_EPOCH, - ); - ctx.bind_role_for_test("writer", info.clone()); - let set = bound::bound_model_set(&prompt, ctx.model_bindings()); - let binding = set.binding("writer").expect("the filled role binds"); + ) + .with_models(BTreeMap::from([(ModelRole::Default, info.clone())])); + let state = state_over(&prompt, ctx); + let set = state.model_set(); + let binding = set.binding("default").expect("the filled role binds"); assert_eq!(binding.render_info(), info); + assert_eq!(state.refusal(), None, "the prompt requires nothing"); } #[test] -fn a_pack_in_both_roster_lists_records_no_vendor() { +fn an_unavailable_roster_pack_records_no_vendor_and_a_pack_listed_twice_keeps_its_last_entry() { let web = PackId::parse("web").expect("a static valid Pack id"); + let usable = RosterEntry::Usable { + vendor: "acme".to_owned(), + }; + let down = RosterEntry::Unavailable { + reason: "the server is down".to_owned(), + }; let state = state_over( &test_prompt(), - RunContext::new( + RunConfig::new( "run-context-test", TEST_SEED, runvil_types::timestamp::Timestamp::UNIX_EPOCH, ) - .with_packs( - vec![UsablePack::new(web.clone(), "acme")], - vec![UnavailablePack::new(web.clone(), "the server is down")], - ), + .with_packs([(web.clone(), usable), (web.clone(), down.clone())]), ); let tool = ToolId::parse("web/fetch").expect("a static valid tool id"); assert_eq!( state.tool_set().vendor_for(&tool), None, - "a name in both lists counts as unavailable, so it has no vendor" + "an unavailable Pack has no vendor" ); assert_eq!( state.tool_set().roster_for_test().get(&web), - Some(&RosterEntry::Unavailable { - reason: "the server is down".to_owned(), - }), - "one roster entry per Pack, and the unavailable one won, carrying the given reason" + Some(&down), + "one roster entry per Pack, the last one given, carrying its reason" ); } diff --git a/crates/runvil-internal/engine/src/execute/context.rs b/crates/runvil-internal/engine/src/execute/context.rs index eb5c8105e..6a2b1fe09 100644 --- a/crates/runvil-internal/engine/src/execute/context.rs +++ b/crates/runvil-internal/engine/src/execute/context.rs @@ -1,10 +1,13 @@ //! The execute subtree's ambient run state. //! -//! [`RunState`] is built once in [`run`](super::run) and travels through -//! the execute subtree as parameter one (`ctx: &RunState`). The -//! invariant: a new run-scoped concern becomes a field here, never a new -//! parameter. Per-call data (a section, a `var` snapshot) stays -//! in parameters or on the per-section frame. +//! [`ChainState`] is built once in [`run`](super::run), as the root +//! chain's state, and travels through the execute subtree as parameter one +//! (`ctx: &ChainState`). It holds the run's one [`SharedState`] and the +//! fields a chain's fork replaces. The invariant: a new run-scoped concern +//! becomes a field here, never a new parameter: a field of +//! [`SharedState`] when it is the same for every chain, and a field of +//! [`ChainState`] when a fork replaces it. Per-call data (a section, a +//! `var` snapshot) stays in parameters or on the per-section frame. use std::fmt; use std::sync::atomic::AtomicU32; @@ -19,10 +22,12 @@ use runvil_types::untrusted::GuardNonce; use runvil_vfs::{Access, VfsRef}; use crate::lua::{Argv, ToolSet}; -use crate::model::{ModelSet, ModelView}; +use crate::model::{ModelBinding, ModelRole, ModelSet}; use crate::parser::Prompt; -use super::config::{RunContext, RunLimits}; +use super::config::{RunConfig, RunLimits}; +use super::fill; +use super::requirements::Requirements; use super::section_context::SectionSys; use super::section_vm::{SectionVmSetup, VmSeed}; @@ -40,20 +45,12 @@ pub(super) struct ReportingHandles { pub(super) turns: Arc, } -/// The ambient state one run shares across the execute subtree. +/// The run-wide state every chain of one run shares: one `Arc` per run, +/// built with the root chain's state and never forked. /// -/// Shared by every chain of the run and cheap to clone: every field is -/// shared ownership or `Copy`, so a clone points at the same run state. -/// Its handles are fixed for the run, though the cancel flag, the event -/// buffer, the turn counter, and the model set's prompt-wide default -/// change behind them. -/// The three sanctioned forks: [`with_walk_state`](Self::with_walk_state) -/// at the H1-to-walk handoff, [`with_task`](Self::with_task) giving a -/// spawned chain its own task emitter and turn counter, and -/// [`with_args`](Self::with_args) passing a `call` or task-spawning arm's -/// input override into its contained chain. -#[derive(Clone)] -pub(crate) struct RunState { +/// Every field is the same for every chain. Its handles are fixed for the +/// run, though the cancel flag and the event buffer change behind them. +struct SharedState { /// The prompt this run executes. prompt: Arc, /// The untrusted-envelope nonce, derived once here from the run's seed @@ -65,39 +62,30 @@ pub(crate) struct RunState { vfs: VfsRef, /// The execution identifier stamped on every observation. execution: Arc, - /// The run's argument string for `{{ args }}` substitution. - args: Arc, - /// The run's `argv`, as `bound::derive_argv` derives it from `args`. At - /// construction this is the value the H1 pass starts from; the walk's - /// fork holds the value H1 left behind at the freeze, so an H1 repair - /// reaches every downstream section. - argv: Option>, /// The run's resource limits. limits: RunLimits, /// The run's event buffer, shared by every chain's emitter, spawned /// task chains' included, and drained once per `step` into the batch /// returned to the caller. events: EventSink, - /// This context's reporting handles: the root task's at construction, - /// a spawned chain's own after [`with_task`](Self::with_task). The Lua - /// layer's seams (`require`'s library loads, `log`, teardown, the - /// shared tool-dispatch body) take the emitter too, so their reports - /// land in the buffer in order with the scheduler's own. - reporting: ReportingHandles, /// The run's cancel flag: polled between chain steps and installed on - /// every section VM's instruction hook. The context's one handle, the - /// same flag the caller watches and the run's `cancel` sets. + /// every section VM's instruction hook. It is the config's one handle, + /// the flag the caller supplies with `RunConfig::with_cancel` and keeps + /// its own clone of. cancel: CancelHandle, - /// The run's tool set: built from the prepared catalog at construction + /// The run's tool set: built from the config's catalog at construction /// and changed by nothing during the run, so every reader, every /// section VM (H1 included) among them, shares it with no lock and no /// copy. tool_set: Arc, - /// The run's model set: built from the prepared bindings at - /// construction and shared with every section VM (H1 included). The - /// only writer is `models.default` (a prompt-wide fact); readers - /// outside the VM layer go through [`models`](Self::models). - model_set: Arc>, + /// The run's model set: filled from the config's role map at + /// construction, shared with every section VM (H1 included) and changed + /// by nothing during the run, so every reader shares it with no lock and + /// no copy. + model_set: Arc, + /// The refusal notice the run's first step fails with: what the prompt + /// requires that the config lacks, or `None` when it lacks nothing. + refusal: Option>, /// The run's `started_at` rendered as RFC 3339, stamped into every /// section's `sys.when`, the H1 pass included. when: Arc, @@ -114,18 +102,48 @@ pub(crate) struct RunState { raw_shims: bool, } -impl RunState { - /// Builds the context for one run of `prompt` over `config`, which it - /// consumes. The turn counter is minted here (starting at zero), as are - /// the run's shared tool and model sets - built from the catalog, the - /// roster, and the model bindings (only the bindings come - /// from [`RunContext::prepare`](super::RunContext::prepare), and they - /// stay empty on a caller-built context that skipped it); the nonce - /// derives from `config`'s seed and `when` renders `config`'s - /// `started_at`, so two contexts over the same inputs agree on both. +/// One chain's state: the run's [`SharedState`] and the fields a chain's +/// fork replaces. +/// +/// Cheap to clone: a clone shares the run's state and this chain's +/// reporting handles, so it advances the same turn counter. +/// The three sanctioned forks: [`with_walk_state`](Self::with_walk_state) +/// at the H1-to-walk handoff, [`with_task`](Self::with_task) giving a +/// spawned chain its own task emitter and turn counter, and +/// [`with_args`](Self::with_args) passing a `call` or task-spawning arm's +/// input override into its contained chain. +#[derive(Clone)] +pub(crate) struct ChainState { + /// The run-wide state every chain of the run shares. + shared: Arc, + /// This context's reporting handles: the root task's at construction, + /// a spawned chain's own after [`with_task`](Self::with_task). The Lua + /// layer's seams (`require`'s library loads, `log`, teardown, the + /// shared tool-dispatch body) take the emitter too, so their reports + /// land in the buffer in order with the scheduler's own. + reporting: ReportingHandles, + /// The run's argument string for `{{ args }}` substitution. + args: Arc, + /// The run's `argv`, as `bound::derive_argv` derives it from `args`. At + /// construction this is the value the H1 pass starts from; the walk's + /// fork holds the value H1 left behind at the freeze, so an H1 repair + /// reaches every downstream section. + argv: Option>, +} + +impl ChainState { + /// Builds the run's shared state and the root chain's state for one + /// run of `prompt` over `config`, which it consumes. The turn counter + /// is minted here (starting at zero), as are the run's shared tool and + /// model sets - the tool set built from the catalog and the roster, + /// the model set filled from the role map (it stays empty for an empty + /// map) - and the refusal notice of what the prompt requires that + /// `config` lacks; the nonce derives from `config`'s seed and `when` + /// renders `config`'s `started_at`, so two states over the same inputs + /// agree on both. #[must_use] - pub(super) fn new(prompt: Arc, args: &str, config: RunContext) -> Self { - let RunContext { + pub(super) fn root(prompt: Arc, args: &str, config: RunConfig) -> Self { + let RunConfig { name, seed, started_at, @@ -135,16 +153,22 @@ impl RunState { limits, ui, vfs, - model_bindings, + models, tools, packs, preludes, - .. } = config; - let tool_set = Arc::new(bound::run_tool_set(&prompt, tools, packs)); - let model_set = Arc::new(Mutex::new(bound::bound_model_set(&prompt, &model_bindings))); + let mut requirements = Requirements::default(); + let model_set = Arc::new(fill::fill_model_set(&prompt, models, &mut requirements)); + requirements.check_packs(&prompt, &tools, packs.as_ref()); + let refusal = requirements.notice().map(Arc::from); + let tool_set = Arc::new(bound::run_tool_set( + &prompt, + tools, + packs.unwrap_or_default(), + )); let execution: Arc = Arc::from(name); - // The root task's event sequence starts at the context's + // The root task's event sequence starts at the config's // `provenance_start`: past any parse events the caller put in the // stream ahead of the run, or at zero. let events = EventSink::seeded(provenance_start); @@ -157,73 +181,85 @@ impl RunState { )); let derived_argv = bound::derive_argv(&prompt, args).map(Arc::from); Self { - prompt, - nonce: GuardNonce::from_seed(&seed), - vfs, - execution, - args: Arc::from(args), - argv: derived_argv, - limits, - events, + shared: Arc::new(SharedState { + prompt, + nonce: GuardNonce::from_seed(&seed), + vfs, + execution, + limits, + events, + cancel, + tool_set, + model_set, + refusal, + when: Arc::from(started_at.to_rfc3339()), + ui: ui.map(Arc::new), + preludes: preludes.into(), + #[cfg(test)] + raw_shims: false, + }), reporting: ReportingHandles { emitter, turns: Arc::new(AtomicU32::new(0)), }, - cancel, - tool_set, - model_set, - when: Arc::from(started_at.to_rfc3339()), - ui: ui.map(Arc::new), - preludes: preludes.into(), - #[cfg(test)] - raw_shims: false, + args: Arc::from(args), + argv: derived_argv, } } + /// The run-wide state, for a test hook that writes it. + /// + /// # Panics + /// Panics once the state has been cloned, because every clone shares + /// the one [`SharedState`]. The hooks run before the run starts, when + /// nothing has cloned it yet. + #[cfg(test)] + fn shared_mut_for_test(&mut self) -> &mut SharedState { + Arc::get_mut(&mut self.shared) + .expect("a test hook writes the run-wide state before anything clones it") + } + /// Exposes the raw `tools.call_as_model` shim in every section VM this /// run starts, so a test fixture can drive the scheduler's `tool_call` /// arm with one model-issued call. #[cfg(test)] pub(crate) fn expose_raw_shims_for_test(&mut self) { - self.raw_shims = true; + self.shared_mut_for_test().raw_shims = true; } - /// Replaces the run's model set with `set`, as the H1 pass would - /// leave it, so a test can skip the pass. Call it before the run - /// starts: every section VM shares the set it finds here. + /// Sets the run's model set to `set`, so a test can run a prompt + /// over bindings it builds by hand. Call it before the run starts: every + /// section VM shares the set it finds here. #[cfg(test)] #[must_use] - pub(crate) fn with_models_for_test(self, set: ModelSet) -> Self { - *self - .model_set - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner) = set; + pub(crate) fn with_models_for_test(mut self, set: ModelSet) -> Self { + self.shared_mut_for_test().model_set = Arc::new(set); self } /// The prompt this run executes. pub(super) fn prompt(&self) -> &Arc { - &self.prompt + &self.shared.prompt } /// The run's cancel flag. pub(super) fn cancel(&self) -> &CancelHandle { - &self.cancel + &self.shared.cancel } /// The run's untrusted-envelope nonce. pub(super) fn nonce(&self) -> &GuardNonce { - &self.nonce + &self.shared.nonce } /// The run's VFS handle. pub(super) fn vfs(&self) -> &VfsRef { - &self.vfs + &self.shared.vfs } /// The execution identifier stamped on every observation. pub(super) fn execution(&self) -> &str { - &self.execution + &self.shared.execution } /// The run's argument string. @@ -239,7 +275,7 @@ impl RunState { /// The run's resource limits. pub(super) fn limits(&self) -> RunLimits { - self.limits + self.shared.limits } /// This context's reporting handles: its task-scoped emitter and its @@ -258,7 +294,7 @@ impl RunState { /// drain, in push order. The run's `step` calls this once per step and /// returns the batch to the caller. pub(super) fn take_events(&self) -> Vec { - self.events.take() + self.shared.events.take() } /// The model-turn counter this context advances. @@ -269,40 +305,48 @@ impl RunState { /// The run's shared tool set, the one every section VM the run /// constructs holds. pub(super) fn tool_set(&self) -> &Arc { - &self.tool_set + &self.shared.tool_set } - /// The run's model set, read-only. - pub(super) fn models(&self) -> &dyn ModelView { - &*self.model_set + /// The binding of the `default` role: the model a section starts on. + /// It is `None` when the caller supplied no model. + pub(super) fn default_binding(&self) -> Option { + self.shared + .model_set + .binding(ModelRole::Default.as_str()) + .cloned() } - /// The run's model set as the concrete handle, shared with every - /// section VM the run constructs. - pub(super) fn model_set(&self) -> Arc> { - Arc::clone(&self.model_set) + /// The run's model set, shared with every section VM the run + /// constructs. + pub(super) fn model_set(&self) -> Arc { + Arc::clone(&self.shared.model_set) + } + + /// The refusal notice the run's first step fails with, or `None` when + /// the config lacks nothing the prompt requires. + pub(super) fn refusal(&self) -> Option<&str> { + self.shared.refusal.as_deref() } /// The resolved tool-loop cap per `models.loop` or model handle `loop` /// call: the frontmatter's `max_tool_iterations` over the limits /// default. fn max_tool_iterations(&self) -> usize { - crate::parser::detail::max_tool_iterations(self.prompt.frontmatter()) - .resolve(self.limits.max_tool_iterations.get() as usize) + crate::parser::detail::max_tool_iterations(self.shared.prompt.frontmatter()) + .resolve(self.shared.limits.max_tool_iterations.get() as usize) } /// The run's top-level section count, reported as `sys.section_count`. fn section_count(&self) -> usize { - crate::parser::detail::sections(&self.prompt).len() + crate::parser::detail::sections(&self.shared.prompt).len() } /// The H1-to-walk handoff: the `argv` H1 left behind at the freeze, - /// set on a cheap clone so the context H1 saw stays untouched. The - /// tool and model sets need no delta: they were built at construction, - /// nothing writes the tool set during the run, and H1's prompt-wide - /// `models.default` landed in the one shared model set. `when` needs - /// none either: it is the run's `started_at`, the same for the pass and - /// the walk. + /// set on a cheap clone so the state H1 saw stays untouched. The + /// run-wide state needs no delta: the tool and model sets were built at + /// construction and nothing writes either during the run, and `when` + /// is the run's `started_at`, the same for the pass and the walk. #[must_use] pub(super) fn with_walk_state(&self, argv: Option) -> Self { let mut ctx = self.clone(); @@ -318,7 +362,7 @@ impl RunState { #[must_use] pub(super) fn with_args(&self, args: &str) -> Self { let mut ctx = self.clone(); - ctx.argv = bound::derive_argv(&self.prompt, args).map(Arc::from); + ctx.argv = bound::derive_argv(&self.shared.prompt, args).map(Arc::from); ctx.args = Arc::from(args); ctx } @@ -341,8 +385,8 @@ impl RunState { /// the run-wide inputs from this context; the constructor supplies only /// its own deltas: the `sys` JSON, the seed, the chain step's access /// capability (the walk's own, a call chain's borrowed parent - /// capability, a task chain's spawned one), the section name, and - /// whether `argv` installs writable. + /// capability, a task chain's spawned one), the section's current-model + /// selection, the section name, and whether `argv` installs writable. /// /// `argv_writable` picks the [`Argv`] mode that /// [`SectionVmSetup::argv`] describes; the H1 pass is the only @@ -352,6 +396,7 @@ impl RunState { sys: &'a serde_json::Value, seed: VmSeed<'a>, access: &'a Arc, + selection: &'a Arc>>, section_name: &'a str, argv_writable: bool, ) -> SectionVmSetup<'a> { @@ -365,14 +410,15 @@ impl RunState { argv, sys, access, + selection, seed, emitter: &self.reporting.emitter, section_name, max_tool_iterations: self.max_tool_iterations(), - ui: self.ui.as_ref(), - preludes: &self.preludes, + ui: self.shared.ui.as_ref(), + preludes: &self.shared.preludes, #[cfg(test)] - raw_shims: self.raw_shims, + raw_shims: self.shared.raw_shims, } } @@ -383,11 +429,11 @@ impl RunState { /// site. pub(super) fn section_sys(&self, id: &str, task_id: &TaskId, section_name: &str) -> SectionSys { SectionSys { - when: Arc::clone(&self.when), + when: Arc::clone(&self.shared.when), id: id.to_owned(), taskid: task_id.clone(), section_name: section_name.to_owned(), - execution: Arc::clone(&self.execution), + execution: Arc::clone(&self.shared.execution), section_count: self.section_count(), index: None, model: None, @@ -395,9 +441,20 @@ impl RunState { } } -impl fmt::Debug for RunState { +impl fmt::Debug for ChainState { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - let mut state = f.debug_struct("RunState"); + f.debug_struct("ChainState") + .field("shared", &self.shared) + .field("reporting", &self.reporting) + .field("args", &self.args) + .field("argv", &self.argv) + .finish() + } +} + +impl fmt::Debug for SharedState { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let mut state = f.debug_struct("SharedState"); #[cfg(test)] state.field("raw_shims", &self.raw_shims); state @@ -405,14 +462,12 @@ impl fmt::Debug for RunState { .field("nonce", &self.nonce) .field("vfs", &"") .field("execution", &self.execution) - .field("args", &self.args) - .field("argv", &self.argv) .field("limits", &self.limits) .field("events", &self.events) - .field("reporting", &self.reporting) .field("cancel", &self.cancel) .field("tool_set", &self.tool_set) .field("model_set", &self.model_set) + .field("refusal", &self.refusal) .field("when", &self.when) .field("ui", &self.ui) .field( diff --git a/crates/runvil-internal/engine/src/execute/error-tests.rs b/crates/runvil-internal/engine/src/execute/error-tests.rs new file mode 100644 index 000000000..b8b8d9eeb --- /dev/null +++ b/crates/runvil-internal/engine/src/execute/error-tests.rs @@ -0,0 +1,47 @@ +//! The stable tag of each run-error kind: the variant's name, which parses +//! back to the kind. + +use super::RunErrorKind; + +/// Every kind, in declaration order. +const KINDS: [RunErrorKind; 10] = [ + RunErrorKind::Parse, + RunErrorKind::Binding, + RunErrorKind::Completion, + RunErrorKind::Tool, + RunErrorKind::Vfs, + RunErrorKind::Lua, + RunErrorKind::Quota, + RunErrorKind::ContextExhausted, + RunErrorKind::Internal, + RunErrorKind::RequirementsUnmet, +]; + +#[test] +fn each_kinds_tag_is_its_debug_name_and_parses_back_to_the_kind() { + for kind in KINDS { + assert_eq!(kind.as_str(), format!("{kind:?}")); + assert_eq!(RunErrorKind::from_tag(kind.as_str()), Some(kind)); + } +} + +#[test] +fn a_retired_tag_parses_to_the_kind_that_absorbed_it_and_no_kind_writes_it() { + for (tag, kind) in [ + ("Version", RunErrorKind::Parse), + ("Determinism", RunErrorKind::Vfs), + ] { + assert_eq!(RunErrorKind::from_tag(tag), Some(kind), "{tag:?}"); + assert!( + KINDS.iter().all(|written| written.as_str() != tag), + "{tag:?}" + ); + } +} + +#[test] +fn a_tag_no_kind_has_parses_to_none() { + for tag in ["Input", "parse", "", "Internal "] { + assert_eq!(RunErrorKind::from_tag(tag), None, "{tag:?}"); + } +} diff --git a/crates/runvil-internal/engine/src/execute/error.rs b/crates/runvil-internal/engine/src/execute/error.rs index 00b61e45e..0c28ffd20 100644 --- a/crates/runvil-internal/engine/src/execute/error.rs +++ b/crates/runvil-internal/engine/src/execute/error.rs @@ -1,13 +1,12 @@ -//! The public run-error surface: [`RunError`], its stable [`RunErrorKind`], -//! and the [`SourceLocation`] that says where a failure happened. - -use std::fmt; -use std::num::NonZeroU32; -use std::ops::Range; +//! The stable [`RunErrorKind`] of a failed run, and the kind and message +//! the Engine's own error ends a run with. use crate::Error; -/// The stable category of a [`RunError`], for matching in code. +use super::RunOutcome; + +/// The stable category of a failed run's [`RunOutcome::Failed`], for matching +/// in code. /// /// Each variant names the condition that failed the run. The enum is /// `#[non_exhaustive]`, so a `match` on it needs a wildcard arm. That lets @@ -17,12 +16,13 @@ use crate::Error; pub enum RunErrorKind { /// Parsing the prompt failed: invalid frontmatter YAML, a failure the /// parser classified as structural, or a Lua block that did not compile. + /// The run's first step reports it too, before any section runs, for a + /// prompt that declares no `runvil:` version or a major version this + /// build does not support. Parse, - /// The prompt declared a `runvil:` major version outside the set - /// this build supports. - Version, - /// Binding a tool or model capability failed, or the capability was - /// absent. + /// Building the tools a section offers its model failed: a tool's + /// model-facing schema did not build, or the section's frozen sets, or + /// the render answer they are merged from, broke a rule of the merge. Binding, /// A model completion failed at the transport, backend, or decode layer. Completion, @@ -31,12 +31,12 @@ pub enum RunErrorKind { Tool, /// A run-scoped store operation failed, or the run lacks a store to /// operate on. - Vfs, - /// Two live executions in the run claimed the same store path. /// - /// The run stops at once to keep the interleaving of their store access - /// deterministic. Lua code cannot catch this failure. - Determinism, + /// It also covers two live executions in the run that claim the same + /// store path. The run then stops at once to keep the interleaving of + /// their store access deterministic, and Lua code cannot catch the + /// failure. + Vfs, /// A section's Lua phase failed to run or return a usable value, or the /// author's program broke where the Engine cannot fix it: a Pack prelude /// that did not load, a chain that ended with tasks still live, a @@ -57,69 +57,69 @@ pub enum RunErrorKind { /// kind, an answer for an id the run never issued, or a second answer /// for one effect. Internal, - /// The environment falls short of what the prompt needs: a missing or - /// unavailable required Pack, a service the caller left out, a model - /// requirement a bound model fails to meet (a context minimum or a hard - /// keyword), a reserved Pack id that the run's tool catalog or Pack - /// roster names, or an H1 block that failed the prompt's hard gate. + /// The environment falls short of what the prompt needs. The run's + /// first step reports it, before any section runs, for a declared Pack + /// that the run's roster lacks or lists as unavailable, a model + /// requirement that the model filling a role fails to meet (a context + /// minimum or a thinking requirement), a non-empty role map with no + /// `default` entry, and a reserved Pack id that the run's tool catalog + /// or Pack roster names. An H1 block that failed the prompt's hard gate + /// reports it too, and so does a `models.infer` or `models.loop` round + /// that finds no current model because the caller supplied none. RequirementsUnmet, } -/// Where a run failed: a position in the prompt source or in the Rust -/// source. -/// -/// The error's [`RunErrorKind`] says which of the two it is, and the -/// extension of `path` says so too. Use the kind to branch in code, the -/// error message to show a reader, and the location to navigate to the -/// fault. Only the Engine builds one. -#[derive(Debug, Clone, PartialEq, Eq)] -#[non_exhaustive] -pub struct SourceLocation { - /// The prompt name or Rust source file where the failure occurred. +impl RunErrorKind { + /// Returns the stable tag for this kind: the variant's name, such as + /// `Parse` or `RequirementsUnmet`. /// - /// For a prompt failure this is the prompt's frontmatter name when - /// parsing got that far. Otherwise it is the placeholder ``, - /// which the caller should replace with its own label for the source. A - /// frontmatter YAML failure always gets the placeholder, because it - /// happens before the name is read. For an internal fault this is the - /// Rust source file, as given by `file!()`. - pub path: String, - /// The 1-based line, when known. - pub line: Option, - /// The 1-based column, when known. - pub column: Option, - /// The byte span of the offending region, when known. - /// - /// Only parse failures can have a span, and a frontmatter YAML failure - /// always has `None`. The offsets index the document body, which - /// excludes the frontmatter and any leading BOM and has CRLF line endings - /// normalized to LF. Use [`line`](SourceLocation::line) and - /// [`column`](SourceLocation::column) to locate the failure in the - /// original file. - pub span: Option>, -} + /// [`RunErrorKind::from_tag`] parses the same word back, so the two + /// round-trip. + #[must_use] + pub fn as_str(self) -> &'static str { + match self { + RunErrorKind::Parse => "Parse", + RunErrorKind::Binding => "Binding", + RunErrorKind::Completion => "Completion", + RunErrorKind::Tool => "Tool", + RunErrorKind::Vfs => "Vfs", + RunErrorKind::Lua => "Lua", + RunErrorKind::Quota => "Quota", + RunErrorKind::ContextExhausted => "ContextExhausted", + RunErrorKind::Internal => "Internal", + RunErrorKind::RequirementsUnmet => "RequirementsUnmet", + } + } -/// The error a failed prompt run reports. -/// -/// A run that fails ends with [`Step::Done`](super::Step::Done), whose -/// result is [`RunResult::Failure`](super::RunResult::Failure) holding this -/// error. Use [`kind`](RunError::kind) to classify the failure in code. The -/// message is self-contained: when the failure keeps an originating error, -/// the message already holds that error's text, and -/// [`originating_error`](RunError::originating_error) returns the error -/// itself. A failure whose message does not hold its underlying error, such -/// as a model round's, keeps it behind [`std::error::Error::source`] instead. -#[derive(Debug)] -pub struct RunError { - inner: Error, + /// Parses a tag from [`RunErrorKind::as_str`] back into its kind. + /// + /// Two tags that older builds wrote also parse, so a stored run keeps + /// its meaning: `Version` reads as [`RunErrorKind::Parse`] and + /// `Determinism` as [`RunErrorKind::Vfs`]. Returns `None` for any + /// other string. + #[must_use] + pub fn from_tag(tag: &str) -> Option { + match tag { + "Parse" | "Version" => Some(RunErrorKind::Parse), + "Binding" => Some(RunErrorKind::Binding), + "Completion" => Some(RunErrorKind::Completion), + "Tool" => Some(RunErrorKind::Tool), + "Vfs" | "Determinism" => Some(RunErrorKind::Vfs), + "Lua" => Some(RunErrorKind::Lua), + "Quota" => Some(RunErrorKind::Quota), + "ContextExhausted" => Some(RunErrorKind::ContextExhausted), + "Internal" => Some(RunErrorKind::Internal), + "RequirementsUnmet" => Some(RunErrorKind::RequirementsUnmet), + _ => None, + } + } } -impl RunError { +impl Error { /// Returns the stable classification of this failure. - #[must_use] - pub fn kind(&self) -> RunErrorKind { - match &self.inner { - Error::Parse(_) => RunErrorKind::Parse, + pub(crate) fn kind(&self) -> RunErrorKind { + match self { + Error::Parse(_) | Error::UnsupportedVersion(_) => RunErrorKind::Parse, Error::LuaQuota { .. } | Error::CallDepthExceeded { .. } => RunErrorKind::Quota, Error::ContextExhausted { .. } => RunErrorKind::ContextExhausted, // A leaked task, a task reached for by a chain that does not @@ -134,106 +134,70 @@ impl RunError { | Error::TaskConsumed { .. } | Error::TaskCancelled { .. } | Error::Substitution(_) => RunErrorKind::Lua, - Error::UnsupportedVersion(_) => RunErrorKind::Version, - Error::RequirementsUnmet { .. } => RunErrorKind::RequirementsUnmet, + Error::RequirementsUnmet { .. } | Error::ModelRequired { .. } => { + RunErrorKind::RequirementsUnmet + } Error::Completion(_) => RunErrorKind::Completion, Error::ToolLoopExhausted | Error::OutOfScopeToolCall { .. } | Error::UnboundToolCall { .. } + | Error::ModelOnlyToolCall { .. } | Error::Tool { .. } => RunErrorKind::Tool, - // A cancelled run ends as `RunResult::Cancelled`, so an + // A cancelled run ends as `RunOutcome::Cancelled`, so an // interruption that reaches a failure is an Engine fault. Error::Internal { .. } | Error::Interrupted => RunErrorKind::Internal, - Error::Store { .. } => RunErrorKind::Vfs, - Error::Determinism(_) => RunErrorKind::Determinism, - Error::BindSchema { .. } | Error::ToolSets { .. } | Error::ModelRequired { .. } => { - RunErrorKind::Binding - } + Error::Store { .. } | Error::Determinism(_) => RunErrorKind::Vfs, + Error::BindSchema { .. } | Error::ToolSets { .. } => RunErrorKind::Binding, } } - /// The Engine's own error behind the boundary one, for a test that - /// matches on its variant. - #[cfg(test)] - pub(crate) fn inner(&self) -> &Error { - &self.inner - } - - /// Returns the error this failure originated from, when it keeps one: a - /// Lua runtime failure keeps the `mlua` error, a Lua compile failure the - /// compiler diagnostic, a schema binding failure the validation error, a - /// tool failure the tool's own error, a store failure the store's - /// structured error, an unmet requirement the failure it stands in for, - /// an internal fault the failure that broke the invariant, and a parse - /// failure the parser's cause. A model round's failure and a failed - /// `{{ }}` substitution answer `None`: whatever cause each holds belongs - /// to the error it wraps. + /// Returns the self-contained text a failed run reports: the error's + /// own text, then each cause down its `source` chain whose text the + /// message does not hold yet, after `: `, then where the failure + /// happened. /// - /// The message already holds the cause's text, so - /// [`std::error::Error::source`] does not return it again, and a printer - /// that walks the source chain shows the text once. - #[must_use] - pub fn originating_error(&self) -> Option<&(dyn std::error::Error + Send + Sync + 'static)> { - self.inner.originating_error() - } - - /// Returns `true` when retrying the run may succeed (transient transport or - /// backend failures). - #[must_use] - pub fn is_retryable(&self) -> bool { - match &self.inner { - Error::Completion(error) => error.is_retryable(), - Error::Parse(_) - | Error::Interrupted - | Error::Lua { .. } - | Error::LuaCompile { .. } - | Error::BindSchema { .. } - | Error::ToolSets { .. } - | Error::Substitution(_) - | Error::ToolLoopExhausted - | Error::TasksLive { .. } - | Error::TaskNotOwned { .. } - | Error::TaskConsumed { .. } - | Error::TaskCancelled { .. } - | Error::OutOfScopeToolCall { .. } - | Error::UnboundToolCall { .. } - | Error::ModelRequired { .. } - | Error::UnsupportedVersion(_) - | Error::RequirementsUnmet { .. } - | Error::Tool { .. } - | Error::Internal { .. } - | Error::LuaQuota { .. } - | Error::CallDepthExceeded { .. } - | Error::ContextExhausted { .. } - | Error::Store { .. } - | Error::Determinism(_) => false, + /// A cause-bearing variant renders its cause in its own text and keeps + /// no `source`, so the chain adds only what a transparent variant's + /// error keeps there. A parse failure ends with its prompt name, line, + /// and column, each when known, as `(prompt {name}, line {line}, + /// column {column})`, and an internal fault with the Rust source file + /// and line of the broken invariant, as `(at {file}:{line})`. + pub(crate) fn message(&self) -> String { + let mut rendered = self.to_string(); + let mut cause = std::error::Error::source(self); + while let Some(current) = cause { + let text = current.to_string(); + if !text.is_empty() && !rendered.contains(&text) { + rendered.push_str(": "); + rendered.push_str(&text); + } + cause = current.source(); + } + if let Some(location) = self.location() { + rendered.push_str(" ("); + rendered.push_str(&location); + rendered.push(')'); } + rendered } - /// Returns where the failure occurred, or `None` when it has no location. - /// - /// A failure of kind `Parse` returns its position in the prompt source. - /// The path is the prompt's frontmatter name when parsing got that far, - /// and the placeholder `` otherwise. The line and column come - /// from the YAML error for a frontmatter failure, or from the span for - /// any other parse failure. A failure of kind `Internal` returns the Rust - /// source file and line of the broken invariant. Failures of every other - /// kind return `None`. - #[must_use] - pub fn location(&self) -> Option { - match &self.inner { - Error::Parse(error) => Some(SourceLocation { - path: error.name().unwrap_or("").to_owned(), - line: error.line().map(NonZeroU32::get), - column: error.column().map(NonZeroU32::get), - span: error.span(), - }), - Error::Internal { file, line, .. } => Some(SourceLocation { - path: (*file).to_owned(), - line: Some(*line), - column: None, - span: None, - }), + /// Where the failure happened, for the end of its message, or `None` + /// when the failure has no location or the parser found none of its + /// parts. + fn location(&self) -> Option { + match self { + Error::Parse(error) => { + let parts: Vec = [ + error.name().map(|name| format!("prompt {name}")), + error.line().map(|line| format!("line {line}")), + error.column().map(|column| format!("column {column}")), + ] + .into_iter() + .flatten() + .collect(); + (!parts.is_empty()).then(|| parts.join(", ")) + } + Error::Internal { file, line, .. } => Some(format!("at {file}:{line}")), Error::Completion(_) | Error::Interrupted | Error::Lua { .. } @@ -248,6 +212,7 @@ impl RunError { | Error::TaskCancelled { .. } | Error::OutOfScopeToolCall { .. } | Error::UnboundToolCall { .. } + | Error::ModelOnlyToolCall { .. } | Error::ModelRequired { .. } | Error::UnsupportedVersion(_) | Error::RequirementsUnmet { .. } @@ -259,22 +224,16 @@ impl RunError { | Error::Determinism(_) => None, } } -} - -impl fmt::Display for RunError { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(f, "{}", self.inner) - } -} -impl std::error::Error for RunError { - fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { - std::error::Error::source(&self.inner) + /// Returns the failed outcome this error ends a run with. + pub(crate) fn to_outcome(&self) -> RunOutcome { + RunOutcome::Failed { + kind: self.kind(), + message: self.message(), + } } } -impl From for RunError { - fn from(inner: Error) -> Self { - RunError { inner } - } -} +#[cfg(test)] +#[path = "error-tests.rs"] +mod tests; diff --git a/crates/runvil-internal/engine/src/execute/fill.rs b/crates/runvil-internal/engine/src/execute/fill.rs index 48199fa84..5db6f4837 100644 --- a/crates/runvil-internal/engine/src/execute/fill.rs +++ b/crates/runvil-internal/engine/src/execute/fill.rs @@ -1,56 +1,96 @@ -//! Prepare's fill function: the trivial model fill. +//! The run's fill function: the caller's role map becomes the run's model +//! set when the run is built. -use crate::model::ThinkingMode; -use crate::parser::{ModelKeyword, Prompt, detail}; +use std::collections::BTreeMap; + +use crate::model::{ModelBinding, ModelInfo, ModelInvocation, ModelRole, ModelSet, ThinkingMode}; +use crate::parser::{Prompt, RoleRequirements}; -use super::bindings::ModelBindings; use super::requirements::{RequirementCheck, Requirements, UnmetRequirement}; -/// The trivial model fill: binds every declared role to the run's -/// current model and checks each role's hard keywords and context -/// minimum against its model info, reporting required versus actual into -/// [`Requirements::unmet_models`]. With no current model there is nothing -/// to fill or check. -pub(super) fn fill_model_bindings( +/// Builds the run's model set from the caller's role map and checks what the +/// prompt declares against the models that fill the roles. +/// +/// A role's model is its own entry in `models`, else the `default` entry. +/// The set holds one binding for each of the five roles, in +/// [`ModelRole::ALL`] order, for every prompt whether or not it declares the +/// role. Each declared role's context minimum and thinking requirement are +/// checked against its model's info, reporting required versus actual into +/// [`Requirements::unmet_models`], and a declared thinking requirement sets +/// the thinking switch on that role's binding. +/// +/// An empty map means the caller has no model, so the set stays empty and +/// nothing is checked. A non-empty map with no `default` entry reports one +/// [`RequirementCheck::Unbound`] entry, and the set stays empty. +pub(super) fn fill_model_set( prompt: &Prompt, - model: Option, + mut models: BTreeMap, requirements: &mut Requirements, -) -> ModelBindings { - let mut bindings = ModelBindings::default(); - let Some(model) = model else { - return bindings; +) -> ModelSet { + let mut set = ModelSet::default(); + if models.is_empty() { + return set; + } + let Some(default) = models.remove(&ModelRole::Default) else { + requirements.unmet_models.push(UnmetRequirement { + role: ModelRole::Default, + check: RequirementCheck::Unbound, + required: "a model".to_owned(), + actual: "none".to_owned(), + }); + return set; }; - for (label, role) in prompt.frontmatter().models() { - if let Some(minimum) = role.min_context() - && model.context() < minimum + for role in ModelRole::ALL { + let info = models.remove(&role).unwrap_or_else(|| default.clone()); + let declared = prompt.frontmatter().role_requirements(role); + let minimum = declared.and_then(RoleRequirements::min_context); + let thinking = declared.and_then(RoleRequirements::thinking); + if let Some(minimum) = minimum + && info.context() < minimum { requirements.unmet_models.push(UnmetRequirement { - role: label.to_owned(), + role, check: RequirementCheck::ContextMinimum, required: minimum.to_string(), - actual: model.context().to_string(), + actual: info.context().to_string(), }); } - for keyword in role.keywords() { - // Soft keywords document author intent; only the hard - // keywords have a model info property to check against. - let actual = match keyword { - ModelKeyword::Thinking if model.thinking() == ThinkingMode::Never => Some("Never"), - ModelKeyword::NoThinking if model.thinking() == ThinkingMode::Always => { - Some("Always") - } - _ => None, - }; - if let Some(actual) = actual { - requirements.unmet_models.push(UnmetRequirement { - role: label.to_owned(), - check: RequirementCheck::HardKeyword, - required: detail::keyword_name(*keyword).to_owned(), - actual: actual.to_owned(), - }); - } + check_thinking(role, thinking, &info, requirements); + let binding = ModelBinding::new( + role.as_str(), + info.description(), + info.id().clone(), + ModelInvocation::new().with_thinking(thinking), + info.context(), + ) + .with_catalog_info(info); + set.push(binding); + } + set +} + +/// Reports the thinking requirement of `role` when `info`'s thinking mode +/// fails it: `thinking: true` needs a model that can think, and +/// `thinking: false` needs one that can run without thinking. +fn check_thinking( + role: ModelRole, + thinking: Option, + info: &ModelInfo, + requirements: &mut Requirements, +) { + let failed = match thinking { + Some(true) if info.thinking() == ThinkingMode::Never => Some(("thinking: true", "Never")), + Some(false) if info.thinking() == ThinkingMode::Always => { + Some(("thinking: false", "Always")) } - bindings.bind(label, model.clone()); + _ => None, + }; + if let Some((required, actual)) = failed { + requirements.unmet_models.push(UnmetRequirement { + role, + check: RequirementCheck::Thinking, + required: required.to_owned(), + actual: actual.to_owned(), + }); } - bindings } diff --git a/crates/runvil-internal/engine/src/execute/infer-tests.rs b/crates/runvil-internal/engine/src/execute/infer-tests.rs index 5844f11fb..ac93b6b3a 100644 --- a/crates/runvil-internal/engine/src/execute/infer-tests.rs +++ b/crates/runvil-internal/engine/src/execute/infer-tests.rs @@ -30,10 +30,9 @@ fn an_infer_round_answered_with_tool_calls_fails_as_a_retryable_malformed_reply( let Error::Completion(failure) = &error else { panic!("required a completion failure, actual {error:?}"); }; - assert_eq!(failure.kind(), CompletionErrorKind::MalformedResponse); - assert!( - failure.is_retryable(), - "a reply that broke protocol may come back well formed" + assert_eq!( + failure.kind(), + CompletionErrorKind::MalformedResponse, + "a reply that broke protocol is malformed, so it may come back well formed" ); - assert!(crate::RunError::from(error).is_retryable()); } diff --git a/crates/runvil-internal/engine/src/execute/infer.rs b/crates/runvil-internal/engine/src/execute/infer.rs index 18a479a3c..56c6b70bf 100644 --- a/crates/runvil-internal/engine/src/execute/infer.rs +++ b/crates/runvil-internal/engine/src/execute/infer.rs @@ -1,8 +1,8 @@ //! The nested-inference round a section's `models.infer` yields resolve to. //! //! One `infer` shape only: a single direct, tool-free model round on a -//! fresh conversation. `h:infer(prompt)` runs it with the handle's -//! frozen binding; `models.infer(prompt)` resolves the section's +//! fresh conversation. `models.infer(model, prompt)` runs it with the +//! handle's frozen binding; `models.infer(prompt)` resolves the section's //! current model and runs the same path. Neither form sends tools, reads or //! changes the section's `messages`, sets `reply`, or touches `sys`, and //! both work inside a loop's tool handlers and compactors, where a loop diff --git a/crates/runvil-internal/engine/src/execute/requirements-tests.rs b/crates/runvil-internal/engine/src/execute/requirements-tests.rs index 86bb45c55..16fe40e23 100644 --- a/crates/runvil-internal/engine/src/execute/requirements-tests.rs +++ b/crates/runvil-internal/engine/src/execute/requirements-tests.rs @@ -1,17 +1,21 @@ -//! Tests for the preflight report's missing services, unavailable -//! Packs, and the reserved Pack ids `adhoc` and `dynamic`. +//! Tests for the run's start checks against its config: each declared +//! Pack against the roster, the reserved Pack ids `adhoc` and `dynamic`, +//! and the order of the refusal notice's lines. The model role checks sit +//! in the execution suite's `prepare` module. -use std::collections::BTreeSet; +use std::collections::BTreeMap; +use std::sync::Arc; +use runvil_types::models::{ModelRole, ThinkingMode}; use runvil_types::packs::PackId; use runvil_types::timestamp::Timestamp; use runvil_types::tools::{ToolCatalog, ToolId, ToolInfo}; use serde_json::json; -use super::{MissingService, Requirements, UnavailablePack, UsablePack}; -use crate::execute::{RunContext, RunErrorKind}; +use crate::execute::{Run, RunConfig, RunErrorKind, RunOutcome, Step}; +use crate::lua::RosterEntry; use crate::parser::Prompt; -use crate::test_support::scenario::TEST_SEED; +use crate::test_support::scenario::{TEST_SEED, test_model_with}; /// The notice line of a run whose catalog or roster names the reserved /// Pack id `reserved`. @@ -26,18 +30,20 @@ fn id(text: &str) -> PackId { PackId::parse(text).expect("a static valid id") } -/// A prompt that declares no Pack and no model. -fn declares_nothing() -> Prompt { - Prompt::parse( - "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n# Run\n\n## Only\n\nDone.\n", - "requirements-test", - ) - .0 - .expect("the requirements test prompt parses") +/// A prompt with `front` added to its frontmatter, whose one section +/// returns `done`. +fn prompt(front: &str) -> Prompt { + let source = format!( + "---\nname: t\ndescription: d\nrunvil: 0\n{front}---\n\n\ + # Run\n\n## Only\n\n```lua\nreturn 'done'\n```\n" + ); + Prompt::parse(&source, "requirements-test") + .0 + .expect("the requirements test prompt parses") } -fn context() -> RunContext { - RunContext::new("requirements-test", TEST_SEED, Timestamp::UNIX_EPOCH) +fn config() -> RunConfig { + RunConfig::new("requirements-test", TEST_SEED, Timestamp::UNIX_EPOCH) } fn catalog(ids: &[&str]) -> ToolCatalog { @@ -51,266 +57,180 @@ fn catalog(ids: &[&str]) -> ToolCatalog { .expect("distinct ids make a valid catalog") } +/// The roster entry of `pack`, usable from `acme`. +fn usable(pack: &str) -> (PackId, RosterEntry) { + ( + id(pack), + RosterEntry::Usable { + vendor: "acme".to_owned(), + }, + ) +} + +/// The roster entry of `pack`, unavailable for `reason`. +fn unavailable(pack: &str, reason: &str) -> (PackId, RosterEntry) { + ( + id(pack), + RosterEntry::Unavailable { + reason: reason.to_owned(), + }, + ) +} + +/// The refusal notice the first step of a run of `prompt` over `ctx` +/// fails with, after asserting the step reported no run events, or +/// `None` when the run passes its start checks and completes. +#[track_caller] +fn refusal(prompt: Prompt, ctx: RunConfig) -> Option { + let mut run = Run::new(Arc::new(prompt), "", ctx); + let Step::Done { outcome, events } = run.step() else { + panic!("the fixture run is done at its first step"); + }; + match outcome { + RunOutcome::Completed { final_text } => { + assert_eq!(final_text, "done"); + None + } + RunOutcome::Failed { kind, message } => { + assert_eq!(kind, RunErrorKind::RequirementsUnmet, "{message}"); + assert!( + events.is_empty(), + "a refused run reports no run events: {events:?}" + ); + Some(message) + } + RunOutcome::Cancelled => panic!("expected a completion or a refusal, got a cancellation"), + } +} + #[test] fn a_catalog_or_roster_naming_a_reserved_pack_id_refuses_the_run_naming_the_id_once() { for reserved in ["adhoc", "dynamic"] { let (grab, put) = (format!("{reserved}/grab"), format!("{reserved}/put")); - let contexts = [ + let configs = [ ( "the catalog", - context().with_tools(catalog(&["web/fetch", &grab, &put])), - ), - ( - "the usable roster", - context().with_packs(vec![UsablePack::new(id(reserved), "acme")], Vec::new()), + config().with_tools(catalog(&["web/fetch", &grab, &put])), ), + ("the usable roster", config().with_packs([usable(reserved)])), ( "the unavailable roster", - context().with_packs( - Vec::new(), - vec![UnavailablePack::new(id(reserved), "the server is down")], - ), + config().with_packs([unavailable(reserved, "the server is down")]), ), ]; - for (named_by, ctx) in contexts { - let (_, requirements) = ctx.prepare(&declares_nothing(), None); + for (named_by, ctx) in configs { assert_eq!( - requirements.reserved, - BTreeSet::from([id(reserved)]), + refusal(prompt(""), ctx), + Some(format!( + "the environment cannot satisfy this prompt:\n{}", + reserved_line(reserved) + )), "{named_by} names {reserved}" ); - assert!(!requirements.is_satisfied(), "{named_by} names {reserved}"); - let refusal = requirements - .refusal() - .expect("a reserved Pack id refuses the run"); - assert_eq!( - refusal.kind(), - RunErrorKind::RequirementsUnmet, - "{named_by}" - ); - let notice = refusal.to_string(); - assert_eq!( - notice.matches(&reserved_line(reserved)).count(), - 1, - "{named_by}: the notice names {reserved} once: {notice}" - ); } } } #[test] -fn a_context_naming_both_reserved_pack_ids_lists_them_sorted_each_once() { - let ctx = context() +fn a_config_naming_both_reserved_pack_ids_lists_them_sorted_each_once() { + // Two catalog tools and the roster all name `dynamic`. + let ctx = config() .with_tools(catalog(&[ "dynamic/search", "web/fetch", "adhoc/grab", "dynamic/call", ])) - .with_packs( - vec![UsablePack::new(id("dynamic"), "acme")], - vec![UnavailablePack::new(id("adhoc"), "the server is down")], - ); - let (_, requirements) = ctx.prepare(&declares_nothing(), None); - assert_eq!( - requirements.reserved, - BTreeSet::from([id("adhoc"), id("dynamic")]) - ); + .with_packs([ + usable("dynamic"), + unavailable("adhoc", "the server is down"), + ]); assert_eq!( - requirements.notice(), - format!( + refusal(prompt(""), ctx), + Some(format!( "the environment cannot satisfy this prompt:\n{}\n{}", reserved_line("adhoc"), reserved_line("dynamic") - ) + )) ); } #[test] -fn a_context_naming_no_reserved_pack_id_leaves_the_reservations_empty() { - let ctx = context() +fn a_config_naming_no_reserved_pack_id_passes_the_start_checks() { + let ctx = config() .with_tools(catalog(&["web/fetch", "dynamics/search", "adhoc2/grab"])) - .with_packs( - vec![UsablePack::new(id("web"), "runvil")], - vec![UnavailablePack::new(id("mcp"), "the server is down")], - ); - let (_, requirements) = ctx.prepare(&declares_nothing(), None); - assert!(requirements.reserved.is_empty(), "{requirements:?}"); - assert!(requirements.is_satisfied(), "{requirements:?}"); + .with_packs([usable("web"), unavailable("mcp", "the server is down")]); + assert_eq!(refusal(prompt(""), ctx), None); } #[test] -fn merge_keeps_the_union_of_both_reports_reserved_ids_sorted_each_once() { - let reserving = |ids: &[&str]| Requirements { - reserved: ids.iter().copied().map(id).collect(), - ..Requirements::default() - }; - let mut kept = reserving(&["dynamic"]); - kept.merge(Requirements::default()); - assert_eq!( - kept.reserved, - BTreeSet::from([id("dynamic")]), - "merging a clear report keeps the ids" - ); - let mut gained = Requirements::default(); - gained.merge(reserving(&["dynamic"])); +fn a_declared_pack_the_roster_lists_as_unavailable_is_refused_with_its_reason_not_as_missing() { + let ctx = config().with_packs([unavailable("web", "the search provider is missing")]); assert_eq!( - gained.reserved, - BTreeSet::from([id("dynamic")]), - "merging a reserving report takes its ids" - ); - let mut union = reserving(&["dynamic"]); - union.merge(reserving(&["dynamic", "adhoc"])); - assert_eq!(union.reserved, BTreeSet::from([id("adhoc"), id("dynamic")])); - assert_eq!( - union.notice(), - format!( - "the environment cannot satisfy this prompt:\n{}\n{}", - reserved_line("adhoc"), - reserved_line("dynamic") + refusal(prompt("uses: [web]\n"), ctx), + Some( + "the environment cannot satisfy this prompt:\n\ + - web is unavailable: the search provider is missing" + .to_owned() ) ); } -fn web_down() -> UnavailablePack { - UnavailablePack::new(id("web"), "the search provider is missing") -} - -#[test] -fn an_unavailable_pack_leaves_the_report_unsatisfied() { - let mut unavailable = Requirements::default(); - unavailable.unavailable_packs.push(web_down()); - assert!(!unavailable.is_satisfied()); - assert!(unavailable.refusal().is_some()); -} - #[test] -fn the_notice_names_an_unavailable_packs_reason() { - let mut requirements = Requirements::default(); - requirements.unavailable_packs.push(web_down()); +fn declared_packs_the_roster_lacks_are_missing_and_list_before_the_unavailable_ones() { + let ctx = config().with_packs([ + unavailable("web", "the search provider is missing"), + unavailable("mcp", "the server is down"), + usable("kit"), + ]); assert_eq!( - requirements.notice(), - "the environment cannot satisfy this prompt:\n\ - - web is unavailable: the search provider is missing" + refusal(prompt("uses: [web, other, kit, mcp, gone]\n"), ctx), + Some( + "the environment cannot satisfy this prompt:\n\ + - missing required Pack: other\n\ + - missing required Pack: gone\n\ + - web is unavailable: the search provider is missing\n\ + - mcp is unavailable: the server is down" + .to_owned() + ), + "each kind of gap lists in declaration order" ); } #[test] -fn merge_drops_a_missing_pack_named_as_unavailable() { - let mut requirements = Requirements::default(); - requirements.missing_packs.push(id("web")); - requirements.missing_packs.push(id("other")); - let mut activation = Requirements::default(); - activation.unavailable_packs.push(web_down()); - requirements.merge(activation); - assert_eq!(requirements.missing_packs, [id("other")]); - assert_eq!(requirements.unavailable_packs, [web_down()]); -} - -#[test] -fn merge_folds_in_unavailable_packs_without_repeating_one() { - let mut requirements = Requirements::default(); - requirements.unavailable_packs.push(web_down()); - let mut other = Requirements::default(); - other.unavailable_packs.push(web_down()); - other - .unavailable_packs - .push(UnavailablePack::new(id("mcp"), "the server is down")); - requirements.merge(other); - assert_eq!( - requirements.unavailable_packs, - [ - web_down(), - UnavailablePack::new(id("mcp"), "the server is down") - ] - ); -} - -fn missing_input(pack: &str) -> MissingService { - MissingService::new(id(pack), "an input broker") -} - -#[test] -fn a_missing_service_leaves_the_report_unsatisfied() { - let mut requirements = Requirements::default(); - assert!(requirements.is_satisfied()); - requirements - .missing_services - .push(missing_input("user-input")); - assert!(!requirements.is_satisfied()); - assert!(requirements.refusal().is_some()); -} - -#[test] -fn merge_folds_in_missing_services_without_repeating_one() { - let mut requirements = Requirements::default(); - requirements - .missing_services - .push(missing_input("user-input")); - let mut activation = Requirements::default(); - activation - .missing_services - .push(missing_input("user-input")); - activation.missing_services.push(missing_input("asker")); - requirements.merge(activation); +fn a_run_given_no_roster_checks_no_declared_pack_and_an_empty_roster_lacks_every_one() { assert_eq!( - requirements.missing_services, - [missing_input("user-input"), missing_input("asker")] + refusal(prompt("uses: [web]\n"), config()), + None, + "a run given no roster checks no declared Pack" ); -} - -#[test] -fn merge_drops_an_incoming_missing_pack_already_named_as_lacking_a_service() { - let mut requirements = Requirements::default(); - requirements - .missing_services - .push(missing_input("user-input")); - let mut other = Requirements::default(); - other.missing_packs.push(id("user-input")); - other.missing_packs.push(id("other")); - requirements.merge(other); - assert_eq!(requirements.missing_packs, [id("other")]); - assert_eq!(requirements.missing_services, [missing_input("user-input")]); -} - -#[test] -fn merge_drops_a_missing_pack_the_incoming_report_names_as_lacking_a_service() { - let mut requirements = Requirements::default(); - requirements.missing_packs.push(id("user-input")); - requirements.missing_packs.push(id("other")); - let mut activation = Requirements::default(); - activation - .missing_services - .push(missing_input("user-input")); - requirements.merge(activation); - assert_eq!(requirements.missing_packs, [id("other")]); - assert_eq!(requirements.missing_services, [missing_input("user-input")]); -} - -#[test] -fn the_notice_names_the_pack_and_the_service_it_lacks() { - let mut requirements = Requirements::default(); - requirements - .missing_services - .push(missing_input("user-input")); assert_eq!( - requirements.notice(), - "the environment cannot satisfy this prompt:\n\ - - user-input needs an input broker, and the environment provides none" + refusal(prompt("uses: [web]\n"), config().with_packs([])), + Some( + "the environment cannot satisfy this prompt:\n- missing required Pack: web".to_owned() + ), + "an empty roster is a roster" ); } #[test] -fn the_notice_lists_missing_services_after_missing_packs() { - let mut requirements = Requirements::default(); - requirements - .missing_services - .push(missing_input("user-input")); - requirements.missing_packs.push(id("web")); - assert_eq!( - requirements.notice(), - "the environment cannot satisfy this prompt:\n\ - - missing required Pack: web\n\ - - user-input needs an input broker, and the environment provides none" - ); +fn the_notice_lists_missing_then_unavailable_packs_then_reserved_ids_then_model_lines() { + let ctx = config() + .with_tools(catalog(&["adhoc/grab"])) + .with_packs([unavailable("kit", "the kit server is down")]) + .with_models(BTreeMap::from([( + ModelRole::Default, + test_model_with("small", 32_000, ThinkingMode::Switchable), + )])); + let reserved = reserved_line("adhoc"); + let expected = [ + "the environment cannot satisfy this prompt:", + "- missing required Pack: web", + "- kit is unavailable: the kit server is down", + &reserved, + "- role 'strong': requires a context of at least 200000 tokens; \ + the model filling it provides 32000", + ]; + let source = "uses: [kit, web]\nmodels: { strong: { min_context: 200000 } }\n"; + assert_eq!(refusal(prompt(source), ctx), Some(expected.join("\n"))); } diff --git a/crates/runvil-internal/engine/src/execute/requirements.rs b/crates/runvil-internal/engine/src/execute/requirements.rs index a5c36b473..56267e644 100644 --- a/crates/runvil-internal/engine/src/execute/requirements.rs +++ b/crates/runvil-internal/engine/src/execute/requirements.rs @@ -1,8 +1,14 @@ -//! The preflight report: [`Requirements`]. +//! What a run's config lacks for its prompt: [`Requirements`], which the +//! run's first step reports as its refusal notice. -use std::collections::BTreeSet; +use std::collections::{BTreeMap, BTreeSet}; +use runvil_types::models::ModelRole; use runvil_types::packs::PackId; +use runvil_types::tools::ToolCatalog; + +use crate::lua::RosterEntry; +use crate::parser::Prompt; #[cfg(test)] #[path = "requirements-tests.rs"] @@ -11,135 +17,79 @@ mod tests; /// The first line of every refusal notice. pub(super) const NOTICE_HEADER: &str = "the environment cannot satisfy this prompt:"; -/// A report of what the caller must still satisfy before a prompt can run. -/// -/// [`RunContext::prepare`](super::RunContext::prepare) returns one -/// together with the prepared context. The report lists only what needs a -/// person's attention. -#[derive(Debug, Clone, Default, PartialEq, Eq)] -#[non_exhaustive] -pub struct Requirements { - /// The model requirements that the bound models fall short of. - /// - /// Each entry names the role, the check that failed, and what was - /// required versus what the model provides. Examples are a - /// `min_context` of 200000 against a model with a 32k context, and - /// `thinking` against a model whose thinking mode is `Never`. Only - /// model binding in `RunContext::prepare` adds these. - pub unmet_models: Vec, - /// The required Packs that the run lacks. - /// - /// The caller adds a Pack the prompt declares that is not installed. - /// The run fails until every one is satisfied. - pub missing_packs: Vec, - /// The required Packs that are installed but need a service - /// the run lacks. - /// - /// There is one entry for each Pack and missing service. The caller - /// adds these and leaves each such Pack's tools out of the catalog. - /// The run fails until the application provides the service. - pub missing_services: Vec, - /// The required Packs that are installed but cannot serve, each - /// with the reason, such as a failure to build. - /// - /// The caller adds these. The run fails until the Pack can serve. - pub unavailable_packs: Vec, +/// What the run's config lacks for its prompt, gathered when the run is +/// built. The run's first step fails with its notice unless it is empty. +#[derive(Debug, Default)] +pub(super) struct Requirements { + /// The model requirements that the models filling the roles fall short + /// of, and the `default` role a non-empty role map leaves unfilled. + pub(super) unmet_models: Vec, + /// The declared Packs the roster lacks, in declaration order. + missing_packs: Vec, + /// The declared Packs the roster lists as unable to serve, each with + /// its reason, in declaration order. + unavailable_packs: Vec<(PackId, String)>, /// The reserved Pack ids, such as `adhoc`, that the run's tool catalog - /// or Pack roster names. - /// - /// The Engine holds the tools under a reserved Pack id itself, so no - /// Pack may take one. - /// [`RunContext::prepare`](super::RunContext::prepare) fills it. The - /// run fails while it holds any id. - pub reserved: BTreeSet, + /// or Pack roster names. The Engine holds the tools under a reserved + /// Pack id itself, so no Pack may take one. + reserved: BTreeSet, } impl Requirements { - /// Returns whether the report lets the run proceed. - /// - /// That holds when every model requirement is met, every required - /// Pack is present, can serve, and has the services it needs, and - /// nothing in the run takes a reserved Pack id. - #[must_use] - pub fn is_satisfied(&self) -> bool { - self.unmet_models.is_empty() - && self.missing_packs.is_empty() - && self.missing_services.is_empty() - && self.unavailable_packs.is_empty() - && self.reserved.is_empty() - } - - /// Adds the entries of `other` to this report. - /// - /// Use it to combine the caller's Pack report with the report - /// from `RunContext::prepare`, so that one refusal names every gap. - /// The merge skips an entry the report already holds, and keeps - /// `reserved` the union of both reports' ids. - /// - /// `unmet_models` is the exception: it is appended whole, so a role the - /// report already lists is listed again rather than skipped. - /// - /// The merge drops the `missing_packs` entry of a Pack that lacks a - /// service or is unavailable: the service or unavailable entry - /// already names the real cause. - pub fn merge(&mut self, other: Requirements) { - push_new(&mut self.missing_packs, other.missing_packs); - push_new(&mut self.missing_services, other.missing_services); - push_new(&mut self.unavailable_packs, other.unavailable_packs); - self.reserved.extend(other.reserved); - let (services, unavailable) = (&self.missing_services, &self.unavailable_packs); - self.missing_packs.retain(|id| { - !services.iter().any(|missing| missing.pack == *id) - && !unavailable.iter().any(|entry| entry.pack == *id) - }); - self.unmet_models.extend(other.unmet_models); - } - - /// Returns the error to fail the run with, or `None` when the report - /// is satisfied. - /// - /// The error is a [`RunError`](super::RunError) of kind - /// [`RequirementsUnmet`](super::RunErrorKind::RequirementsUnmet) that - /// reports the [`notice`](Requirements::notice). The caller checks it - /// after merging every report into one, and before building the run. - #[must_use] - pub fn refusal(&self) -> Option { - (!self.is_satisfied()).then(|| { - super::RunError::from(crate::Error::RequirementsUnmet { - notice: self.notice(), - cause: None, - }) - }) + /// Checks each Pack `prompt` declares against `roster`, when the run + /// has one, and records each reserved Pack id that a tool of `catalog` + /// or a Pack of `roster`, usable or not, has. + pub(super) fn check_packs( + &mut self, + prompt: &Prompt, + catalog: &ToolCatalog, + roster: Option<&BTreeMap>, + ) { + if let Some(roster) = roster { + for entry in prompt.frontmatter().uses() { + let id = entry.id().clone(); + match roster.get(&id) { + None => self.missing_packs.push(id), + Some(RosterEntry::Unavailable { reason }) => { + self.unavailable_packs.push((id, reason.clone())); + } + Some(RosterEntry::Usable { .. }) => {} + } + } + } + self.reserved = catalog + .tools() + .iter() + .map(|tool| tool.id.pack()) + .chain(roster.into_iter().flat_map(BTreeMap::keys).cloned()) + .filter(PackId::is_reserved) + .collect(); } - /// Returns the refusal notice, which explains what blocks the run. + /// Returns the refusal notice, which explains what blocks the run, or + /// `None` when nothing does. /// /// Each line names what is missing or falls short. The model lines /// carry required versus actual; the Pack lines name the Pack and, for - /// a service or an unavailable Pack, the reason. The notice is written - /// for a model to read: concise, factual, and self-contained. - #[must_use] - pub fn notice(&self) -> String { + /// an unavailable Pack, the reason. The notice is written for a model + /// to read: concise, factual, and self-contained. + pub(super) fn notice(&self) -> Option { // Writing to a String is infallible, so each `write!` result is // discarded under the denied `unwrap_used`/`expect_used` lints. use std::fmt::Write as _; + if self.unmet_models.is_empty() + && self.missing_packs.is_empty() + && self.unavailable_packs.is_empty() + && self.reserved.is_empty() + { + return None; + } let mut notice = String::from(NOTICE_HEADER); for id in &self.missing_packs { let _ = write!(notice, "\n- missing required Pack: {id}"); } - for missing in &self.missing_services { - let _ = write!( - notice, - "\n- {} needs {}, and the environment provides none", - missing.pack, missing.service - ); - } - for entry in &self.unavailable_packs { - let _ = write!( - notice, - "\n- {} is unavailable: {}", - entry.pack, entry.reason - ); + for (id, reason) in &self.unavailable_packs { + let _ = write!(notice, "\n- {id} is unavailable: {reason}"); } for id in &self.reserved { let _ = write!( @@ -149,155 +99,60 @@ impl Requirements { ); } for unmet in &self.unmet_models { + let role = unmet.role.as_str(); let _ = match unmet.check { RequirementCheck::ContextMinimum => write!( notice, - "\n- role '{}': requires a context of at least {} tokens; \ - the current model provides {}", - unmet.role, unmet.required, unmet.actual + "\n- role '{role}': requires a context of at least {} tokens; \ + the model filling it provides {}", + unmet.required, unmet.actual + ), + RequirementCheck::Thinking => write!( + notice, + "\n- role '{role}': requires '{}'; \ + the thinking capability of the model filling it is {}", + unmet.required, unmet.actual ), - RequirementCheck::HardKeyword => write!( + RequirementCheck::Unbound => write!( notice, - "\n- role '{}': requires '{}'; \ - the current model's thinking capability is {}", - unmet.role, unmet.required, unmet.actual + "\n- role '{role}': requires {}; \ + the caller's role map has no '{role}' entry", + unmet.required ), }; } - notice - } -} - -/// Appends each entry of `incoming` that `entries` does not already hold. -fn push_new(entries: &mut Vec, incoming: Vec) { - for entry in incoming { - if !entries.contains(&entry) { - entries.push(entry); - } - } -} - -/// An installed Pack that can serve the run, and its vendor. -/// -/// The run's roster, set with -/// [`RunContext::with_packs`](super::RunContext::with_packs), lists every -/// one the run holds. The caller reports each Pack's vendor as it picked the -/// Pack, and every tool-call and tool-result event for one of the Pack's -/// tools records it. -/// -/// The vendor promise holds only for a Pack the roster lists as usable. A -/// Pack named in both roster lists counts as unavailable, so it has no -/// vendor and its tools record none. -#[derive(Debug, Clone, PartialEq, Eq)] -#[non_exhaustive] -pub struct UsablePack { - /// The installed Pack. - pub pack: PackId, - /// The vendor the caller picked the Pack from, such as `runvil` or - /// `acme`. - pub vendor: String, -} - -impl UsablePack { - /// Creates an entry stating that `pack`, from `vendor`, can serve. - /// - /// Only the caller builds these, for its roster. - #[must_use] - pub fn new(pack: PackId, vendor: impl Into) -> UsablePack { - UsablePack { - pack, - vendor: vendor.into(), - } - } -} - -/// An installed Pack that cannot serve the run, and why. -/// -/// The caller's Pack report lists each one the prompt declares, and the -/// run's roster, set with -/// [`RunContext::with_packs`](super::RunContext::with_packs), lists every -/// one the run holds. -#[derive(Debug, Clone, PartialEq, Eq)] -#[non_exhaustive] -pub struct UnavailablePack { - /// The installed Pack. - pub pack: PackId, - /// Why it cannot serve, written for the model that reads the refusal. - pub reason: String, -} - -impl UnavailablePack { - /// Creates an entry stating that `pack` cannot serve, because of - /// `reason`. - /// - /// Only the caller builds these, for its Pack report and its roster. - #[must_use] - pub fn new(pack: PackId, reason: impl Into) -> UnavailablePack { - UnavailablePack { - pack, - reason: reason.into(), - } - } -} - -/// A service that a required Pack needs and the application lacks. -/// -/// For example, a Pack that asks the operator a question needs a -/// service that reaches the operator, and a batch application runs on its -/// own, so it lacks that service. -#[derive(Debug, Clone, PartialEq, Eq)] -#[non_exhaustive] -pub struct MissingService { - /// The installed, required Pack that needs the service. - pub pack: PackId, - /// The name of the missing service. - /// - /// The caller defines its own service names, so the report carries - /// the name verbatim. - pub service: String, -} - -impl MissingService { - /// Creates an entry stating that `pack` needs the service named - /// `service`, which the application lacks. - /// - /// Only the caller's Pack report holds these. - #[must_use] - pub fn new(pack: PackId, service: impl Into) -> MissingService { - MissingService { - pack, - service: service.into(), - } + Some(notice) } } -/// A model requirement that the model bound to a role falls short of. -/// -/// It names the role, the check that failed, and what the prompt required -/// versus what the bound model provides. -#[derive(Debug, Clone, PartialEq, Eq)] -#[non_exhaustive] -pub struct UnmetRequirement { - /// The declared role label whose requirement failed. - pub role: String, +/// A model requirement that the model filling a role falls short of, or a +/// role the caller left without a model: the role, the check that failed, +/// and what the prompt required versus what the caller provided. +#[derive(Debug)] +pub(super) struct UnmetRequirement { + /// The role whose requirement failed. + pub(super) role: ModelRole, /// Which requirement check failed. - pub check: RequirementCheck, - /// What the prompt required, such as a context minimum of `200000` or - /// the `thinking` keyword. - pub required: String, - /// What the bound model provides, such as a context of `32000` or a - /// thinking mode of `Never`. - pub actual: String, + pub(super) check: RequirementCheck, + /// What the prompt required, such as a context minimum of `200000`, a + /// thinking switch of `thinking: true`, or `a model`. + pub(super) required: String, + /// What the filling model provides, such as a context of `32000` or a + /// thinking mode of `Never`, or `none` when the caller gave the role no + /// model. + pub(super) actual: String, } /// Which model requirement check failed. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -#[non_exhaustive] -pub enum RequirementCheck { - /// The role's context minimum exceeds the context of the model bound - /// to the role. +#[derive(Debug, Clone, Copy)] +pub(super) enum RequirementCheck { + /// The role's context minimum exceeds the context of the model that + /// fills the role. ContextMinimum, - /// The role declares a hard keyword, `thinking` or `no-thinking`, that - /// the bound model's info fails to satisfy. - HardKeyword, + /// The role's thinking requirement, `thinking: true` or + /// `thinking: false`, is one the filling model's info fails to satisfy. + Thinking, + /// The caller's role map holds entries but none for the `default` + /// role, which every role without its own entry falls back to. + Unbound, } diff --git a/crates/runvil-internal/engine/src/execute/run.rs b/crates/runvil-internal/engine/src/execute/run.rs index c732cecfa..60a9e8a8b 100644 --- a/crates/runvil-internal/engine/src/execute/run.rs +++ b/crates/runvil-internal/engine/src/execute/run.rs @@ -4,32 +4,29 @@ //! `cancel` modules. //! //! The effect vocabulary - [`Effect`] with its [`Round`] and -//! [`PackToolSets`], [`EffectAnswer`], [`EffectId`], the serializable -//! records of effects and answers, and the [`ToolCallOrigin`] and -//! [`ToolCaller`] it re-exports from `runvil-types` - comes from the -//! `effect` child module and is re-exported here. +//! [`PackToolSets`], [`EffectAnswer`], [`EffectId`], and the +//! [`ToolCallOrigin`] and [`ToolCaller`] it re-exports from +//! `runvil-types` - comes from the `effect` child module and is +//! re-exported here. The records of effects and answers stay in that +//! module; the caller reads them as JSON. use std::fmt; use std::sync::Arc; -use runvil_types::cancel::CancelHandle; use runvil_types::event::Event; use runvil_types::ids::Provenance; mod effect; -pub use effect::{ - AnswerRecord, ChatAnswerRecord, Effect, EffectAnswer, EffectId, EffectRecord, FailureRecord, - PackToolSets, Round, ToolAnswerRecord, ToolCallOrigin, ToolCaller, -}; +#[cfg(test)] +pub(crate) use effect::{AnswerRecord, ChatAnswerRecord, EffectRecord}; +pub use effect::{Effect, EffectAnswer, EffectId, PackToolSets, Round, ToolCallOrigin, ToolCaller}; -use crate::parser::{ParseErrorKind, Prompt}; -use crate::{Error, Result}; +use crate::parser::Prompt; -use super::RunResult; -use super::config::RunContext; -use super::context::RunState; -use super::error::RunError; +use super::RunOutcome; +use super::config::RunConfig; +use super::context::ChainState; use super::scheduler::Scheduler; /// The outcome of one call to [`Run::step`]: the run either continues or @@ -48,23 +45,27 @@ pub enum Step { effects: Vec, /// The events the step reported, in order. events: Vec, - /// The tool-call effects this step orphaned, in effect-id order. - /// Their answers will be discarded, so the caller may drop each one - /// it still holds; it still owes each one answer. Empty once the - /// run is decided, when the caller may drop everything it holds. - /// It may name a call that `effects` lists, because one step can - /// issue a call and then orphan it. + /// The effects this step withdrew, in effect-id order. Their + /// answers will be discarded, so the caller may drop each one + /// unperformed, or abort it, and answer it + /// [`EffectAnswer::Dropped`]; it still owes each one answer. A step + /// withdraws a tool call whose chain stopped waiting on it. The + /// step that decides the run's outcome withdraws every effect still + /// outstanding, of every kind, so a caller that follows this list + /// drops everything it holds for the run. It may name an effect + /// that `effects` lists, because one step can issue an effect and + /// then withdraw it. withdrawn: Vec, }, /// The run is over. /// - /// Carries the run's result and its last events. A step returns `Done` + /// Carries the run's outcome and its last events. A step returns `Done` /// only after every issued effect has been answered. The event that /// marks the run's end comes with `Done`, or with an earlier `Pending` /// step when effects were still outstanding as the run ended. Done { /// The run's outcome. - result: RunResult, + outcome: RunOutcome, /// The events reported since the previous step. events: Vec, }, @@ -84,73 +85,54 @@ pub struct IssuedEffect { } /// One run of one prompt, which the caller drives through -/// [`step`](Self::step) and [`resume`](Self::resume). +/// [`step`](Self::step) and [`resume`](Self::resume), and cancels through +/// the flag it supplies with +/// [`RunConfig::with_cancel`](super::RunConfig::with_cancel). /// /// `Run` is `Send`: one caller drives it at a time, and the thread may /// change between calls. It holds its prompt through an `Arc`, so the /// caller can parse a prompt once and run it many times. pub struct Run { - /// Whether the run is live, failed to start, or has reported that - /// failure. - start: Start, - /// The run's cancel flag, shared with every chain's VM hook. - cancel: CancelHandle, -} - -/// Where a run stands: live with its scheduler, failed to start with the -/// failure its first step reports, or spent once that step has reported -/// it. -enum Start { - /// The run started; its scheduler steps it. - Live(Box), - /// The run failed to start; the first step reports this failure. - Stillborn(Error), - /// The run failed to start and its first step reported why. - Spent, + /// The scheduler that steps every chain of the run. + scheduler: Box, } impl fmt::Debug for Run { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - let start = match &self.start { - Start::Live(_) => "live", - Start::Stillborn(_) => "stillborn", - Start::Spent => "spent", - }; - let mut state = f.debug_struct("Run"); - state.field("start", &start); - if let Start::Stillborn(error) = &self.start { - state.field("failure", error); - } - state.field("cancel", &self.cancel).finish() + f.debug_struct("Run") + .field("phase", &self.scheduler.phase_name()) + .field("cancel", self.scheduler.cancel_handle()) + .finish() } } impl Run { - /// Builds a run of `prompt` with the arguments `args` and the context + /// Builds a run of `prompt` with the arguments `args` and the config /// `config`. /// - /// A context that skipped - /// [`RunContext::prepare`](super::RunContext::prepare) runs with an - /// empty model set. Its tool set holds whatever catalog and roster the - /// context's builders set. + /// Building a run fills its five model roles from the config's role + /// map, as [`RunConfig::with_models`](super::RunConfig::with_models) + /// describes, and an empty map leaves the model set empty. Its tool set + /// holds whatever catalog and roster the config's builders set. /// - /// A run that fails to start is still built, and its first `step` - /// returns `Done` with the failure. A run fails to start when the - /// prompt has no supported `runvil:` version, when the context's - /// filesystem declares no store, or when the store's backend fails its - /// probe. + /// Building a run never fails. The first `step` checks, in order, + /// that the prompt declares a supported `runvil:` version, that the + /// config's filesystem declares a store whose backend passes its + /// probe, and that the config meets what the prompt requires: each + /// Pack the prompt declares is on the roster and can serve (a run given + /// no roster skips this check), each declared role's requirements hold + /// for the model that fills it, a non-empty role map has a `default` + /// entry, and no tool of the catalog and no Pack of the roster has a + /// Pack id the Engine reserves, such as `adhoc`. When a check fails, + /// that step returns `Done` with the failure and no events. The + /// requirements failure is a + /// [`RequirementsUnmet`](super::RunErrorKind::RequirementsUnmet) whose + /// message names every gap. When the run's cancel flag is set as a + /// check fails, the run ends [`Cancelled`](RunOutcome::Cancelled) + /// instead. #[must_use] - pub fn new(prompt: Arc, args: &str, config: RunContext) -> Run { - // The context's one flag, held here so a stillborn run still has - // the handle `cancel` and `cancel_handle` name. - let cancel = config.cancel.clone(); - match prepare_state(prompt, args, config) { - Ok(state) => Self::from_state(state), - Err(error) => Run { - start: Start::Stillborn(error), - cancel, - }, - } + pub fn new(prompt: Arc, args: &str, config: RunConfig) -> Run { + Self::from_state(ChainState::root(prompt, args, config)) } /// [`Run::new`], with `edit` applied to the assembled state before the @@ -160,26 +142,17 @@ impl Run { pub(crate) fn new_for_test( prompt: Arc, args: &str, - config: RunContext, - edit: impl FnOnce(RunState) -> RunState, + config: RunConfig, + edit: impl FnOnce(ChainState) -> ChainState, ) -> Run { - let cancel = config.cancel.clone(); - match prepare_state(prompt, args, config) { - Ok(state) => Self::from_state(edit(state)), - Err(error) => Run { - start: Start::Stillborn(error), - cancel, - }, - } + Self::from_state(edit(ChainState::root(prompt, args, config))) } - /// Builds the live run over a state that passed `prepare_state`. + /// Builds the run over its assembled state. #[must_use] - fn from_state(state: RunState) -> Run { - let cancel = state.cancel().clone(); + fn from_state(state: ChainState) -> Run { Run { - start: Start::Live(Box::new(Scheduler::new(state))), - cancel, + scheduler: Box::new(Scheduler::new(state)), } } @@ -191,23 +164,14 @@ impl Run { /// answered. Calling `step` after `Done` is a caller error: it returns /// `Done` again with an internal failure. /// + /// The first call checks that the run can start, as + /// [`new`](Self::new) describes. + /// /// `step` cannot fail: a run's failures are values in - /// [`RunResult::Failure`]. So the caller drives the loop and owns the + /// [`RunOutcome::Failed`]. So the caller drives the loop and owns the /// retry policy without catching a panic. pub fn step(&mut self) -> Step { - if let Start::Live(scheduler) = &mut self.start { - return scheduler.step(); - } - let failure = match std::mem::replace(&mut self.start, Start::Spent) { - Start::Stillborn(error) => error, - Start::Live(_) | Start::Spent => { - Error::internal("a run that failed to start cannot be stepped again") - } - }; - Step::Done { - result: RunResult::Failure(RunError::from(failure)), - events: Vec::new(), - } + self.scheduler.step() } /// Delivers the answer to one effect the run issued. @@ -224,102 +188,39 @@ impl Run { /// answer for an effect whose chain stopped waiting is discarded. /// /// `resume` cannot fail: a bad answer ends the run with a - /// [`RunResult::Failure`] instead of returning an error or panicking. + /// [`RunOutcome::Failed`] instead of returning an error or panicking. pub fn resume(&mut self, id: EffectId, answer: EffectAnswer) { - if let Start::Live(scheduler) = &mut self.start { - scheduler.resume(id, answer); - } - } - - /// Asks the run to stop by setting its cancel flag. - /// - /// Running Lua code sees the flag through its instruction hook. The - /// next `step` tears down every chain. The run then reports itself as - /// cancelled once every outstanding effect is answered. - /// - /// Cancellation is a request that later steps complete. The caller - /// must answer each effect it abandons with [`EffectAnswer::Dropped`]. - /// Like `step` and `resume`, `cancel` cannot fail. - pub fn cancel(&mut self) { - self.cancel.cancel(); + self.scheduler.resume(id, answer); } - /// Returns the run's cancel flag, so the caller can cancel the run from - /// another thread. - #[must_use] - pub fn cancel_handle(&self) -> CancelHandle { - self.cancel.clone() - } - - /// Returns whether the run's outcome is decided. - /// - /// The outcome is decided once the run has reported its end event, or - /// when the run failed to start. Any effect still outstanding at that - /// point is an orphan: only `Done` waits for its answer. The caller can - /// read this after a `Pending` step to learn that it may drop what it - /// holds for the run. This keeps control flow independent of the - /// events, which are only a report. - #[must_use] - pub fn decided(&self) -> bool { - match &self.start { - Start::Live(scheduler) => scheduler.decided(), - Start::Stillborn(_) | Start::Spent => true, - } + /// Whether the run's outcome is decided: it has reported its end + /// event, or found at its first step that it cannot start. For the + /// suites that pin when a run is decided apart from what its steps + /// return. + #[cfg(test)] + pub(crate) fn decided_for_test(&self) -> bool { + self.scheduler.decided() } /// The scheduler behind the run, for the suites that inspect its /// arena. #[cfg(test)] pub(crate) fn scheduler_for_test(&mut self) -> &mut Scheduler { - match &mut self.start { - Start::Live(scheduler) => scheduler, - Start::Stillborn(_) | Start::Spent => { - panic!("a run built from a state holds its scheduler") - } - } + &mut self.scheduler } - /// The run's model-turn counter, for the suites that count turns: zero - /// for a run that failed to start. + /// The run's model-turn counter, for the suites that count turns. #[cfg(test)] pub(crate) fn turns_for_test(&self) -> u32 { - match &self.start { - Start::Live(scheduler) => scheduler.turns_for_test(), - Start::Stillborn(_) | Start::Spent => 0, - } + self.scheduler.turns_for_test() } -} -/// Assembles the run state from the caller's context, checking the version -/// gate and then the declared store. -/// -/// # Errors -/// Returns [`Error::UnsupportedVersion`] or a structural parse error for a -/// prompt that is not a supported runvil prompt, or [`Error::Store`] -/// when the handle declares no store or the store's backend fails the -/// probe. -fn prepare_state(prompt: Arc, args: &str, config: RunContext) -> Result { - match prompt.frontmatter().runvil() { - Some(0) => {} - Some(other) => return Err(Error::UnsupportedVersion(other)), - None => { - return Err(Error::parse( - ParseErrorKind::Structure, - "not a runvil prompt: no runvil version", - ) - .with_prompt_name(prompt.frontmatter().name())); - } + /// Takes the Engine error behind the failed outcome the last `Done` + /// carried, for the suites that match on its variant. + #[cfg(test)] + pub(crate) fn take_failure_for_test(&mut self) -> Option { + self.scheduler.take_failure_for_test() } - // Every run and every store call needs a declared store: the probe - // stats the store root through a throwaway capability, so a handle - // with no declaration fails here rather than on the first store call, - // and a declared-but-failing backend's error fails the run. - let probe = config - .vfs - .acquire(runvil_vfs::Origin::new("store mount probe")) - .map_err(Error::store)?; - runvil_vfs::detail::probe_store(&probe).map_err(Error::store)?; - Ok(RunState::new(prompt, args, config)) } #[cfg(test)] diff --git a/crates/runvil-internal/engine/src/execute/run/effect-record.rs b/crates/runvil-internal/engine/src/execute/run/effect-record.rs index 104ae609c..0cdbae799 100644 --- a/crates/runvil-internal/engine/src/execute/run/effect-record.rs +++ b/crates/runvil-internal/engine/src/execute/run/effect-record.rs @@ -58,8 +58,7 @@ impl PackToolSets { /// caller may serve the slot with a different model from the bound one. /// The answer's [`ChatAnswerRecord`] names the model that served the round. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] -#[non_exhaustive] -pub enum EffectRecord { +pub(crate) enum EffectRecord { /// One model round. Chat { /// The round's run-wide id, which the round's content events also @@ -127,8 +126,7 @@ pub enum EffectRecord { /// variant per answer kind, each holding its outcome with every failure /// recorded as a [`FailureRecord`]. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] -#[non_exhaustive] -pub enum AnswerRecord { +pub(crate) enum AnswerRecord { /// The model round's outcome. Chat(std::result::Result), /// Each Pack's rendered tools per set, in the order asked, whole, @@ -156,12 +154,11 @@ pub enum AnswerRecord { /// the `VfsError` tag for a store operation (such as `not_found`). /// `message` is the failure's display text. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] -#[non_exhaustive] -pub struct FailureRecord { +pub(crate) struct FailureRecord { /// The stable tag of the failure's kind. - pub kind: String, + pub(crate) kind: String, /// The failure's display text. - pub message: String, + pub(crate) message: String, } impl FailureRecord { @@ -180,17 +177,16 @@ impl FailureRecord { /// answer. The round's request and response bodies travel separately, as /// debug events. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] -#[non_exhaustive] -pub struct ChatAnswerRecord { +pub(crate) struct ChatAnswerRecord { /// The model that served the round, as the completion names it. - pub model: String, + pub(crate) model: String, /// The provider's finish reason, when it sent one. - pub finish_reason: Option, + pub(crate) finish_reason: Option, /// The reply text, when the round produced text. - pub reply: Option, + pub(crate) reply: Option, /// The names of the tools the model requested, in call order, when it /// requested any. - pub tool_calls: Vec, + pub(crate) tool_calls: Vec, } #[cfg(test)] @@ -252,12 +248,11 @@ impl From<&Completion> for ChatAnswerRecord { /// A tool's own output as the log records it. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] -#[non_exhaustive] -pub struct ToolAnswerRecord { +pub(crate) struct ToolAnswerRecord { /// The output text, before the Engine's trust rules apply. - pub text: String, + pub(crate) text: String, /// The trust level the tool declared for its output. - pub trust: OutputTrust, + pub(crate) trust: OutputTrust, } impl ToolAnswerRecord { diff --git a/crates/runvil-internal/engine/src/execute/run/effect-tests.rs b/crates/runvil-internal/engine/src/execute/run/effect-tests.rs index 76bf1eb4a..c60be26f1 100644 --- a/crates/runvil-internal/engine/src/execute/run/effect-tests.rs +++ b/crates/runvil-internal/engine/src/execute/run/effect-tests.rs @@ -7,13 +7,17 @@ //! rendered tools whole; a `ToolCall` answer projects to its text and //! trust; and every failure projects to its kind's tag and its display //! text. Each record round-trips through serde, as a run log and a replay -//! depend on, and a record written before the render existed reads -//! unchanged. +//! depend on, the public JSON record of every kind is the typed record's +//! JSON and reads back identical, and a record written before the render +//! existed reads unchanged. + +use std::fmt; use runvil_model_client::client::{CompletionResult, RawExchange, ToolCall}; use runvil_model_client::model::CompletionErrorKind; use runvil_types::packs::PackId; use runvil_types::tools::{OutputTrust, ToolErrorKind}; +use serde::de::DeserializeOwned; use serde_json::json; use super::*; @@ -51,7 +55,7 @@ fn three_messages() -> Vec { #[test] fn a_chat_record_copies_after_and_keep_and_logs_only_the_messages_after_keep() { let effect = chat_effect(three_messages(), Some((RoundId::new(2), 2))); - let record = effect.record(); + let record = effect.to_record(); let EffectRecord::Chat { round, after, @@ -93,7 +97,7 @@ fn a_chat_record_copies_after_and_keep_and_logs_only_the_messages_after_keep() { #[test] fn a_keep_equal_to_the_message_count_logs_no_messages() { - let record = chat_effect(three_messages(), Some((RoundId::new(2), 3))).record(); + let record = chat_effect(three_messages(), Some((RoundId::new(2), 3))).to_record(); let EffectRecord::Chat { after, keep, @@ -134,7 +138,7 @@ fn a_render_record_keeps_the_slots_alias_the_entry_and_the_offered_ids() { binding: test_binding("writer", "A general model for tests", "test-model"), offered: tools_pack("Echo the value.", "echo"), }; - let record = effect.record(); + let record = effect.to_record(); let canonical = serde_json::to_value(&record).expect("a render record converts"); assert_eq!( canonical.to_string(), @@ -147,11 +151,11 @@ fn a_render_record_keeps_the_slots_alias_the_entry_and_the_offered_ids() { #[test] fn a_render_answer_records_each_rendered_tool_whole() { - let record = EffectAnswer::Render(tools_pack("Shout the value.", "shout")).record(); + let record = EffectAnswer::Render(tools_pack("Shout the value.", "shout")).to_record(); let canonical = serde_json::to_value(&record).expect("a render answer record converts"); assert_eq!( canonical.to_string(), - r#"{"Render":[{"dynamic_set":[],"pack":"tools","static_set":[{"describes_tools":false,"description":"Shout the value.","id":"tools/echo","parameters_schema":{"type":"object"},"structured_output":false,"survives_stop":false,"wire_name":"shout"}]}]}"# + r#"{"Render":[{"dynamic_set":[],"pack":"tools","static_set":[{"describes_tools":false,"description":"Shout the value.","id":"tools/echo","model_only_call":false,"parameters_schema":{"type":"object"},"structured_output":false,"survives_stop":false,"wire_name":"shout"}]}]}"# ); let back: AnswerRecord = serde_json::from_value(canonical).expect("the canonical form reads back"); @@ -180,7 +184,7 @@ fn records_written_before_the_render_existed_read_unchanged() { #[test] fn a_first_send_records_no_after_and_logs_every_message() { - let record = chat_effect(three_messages(), None).record(); + let record = chat_effect(three_messages(), None).to_record(); let EffectRecord::Chat { after, keep, @@ -213,7 +217,7 @@ fn a_chat_answer_records_the_reply_model_and_finish_reason_without_the_bodies() CompletionResult::Text("the reply".to_owned()), "stop", )))); - let record = answer.record(); + let record = answer.to_record(); assert_eq!( record, AnswerRecord::Chat(Ok(ChatAnswerRecord::new("test-model") @@ -235,7 +239,7 @@ fn a_chat_answer_with_tool_calls_records_their_names_in_call_order_and_no_reply( ToolCall::from_parts("call-2", "echo", json!({})).expect("a whole call"), ]); let answer = EffectAnswer::Chat(Ok(Box::new(finished_completion(calls, "tool_calls")))); - let record = answer.record(); + let record = answer.to_record(); assert_eq!( record, AnswerRecord::Chat(Ok(ChatAnswerRecord::new("test-model") @@ -257,7 +261,7 @@ fn a_canned_chat_answer_records_no_finish_reason() { .expect("a text result is accepted"); let answer = EffectAnswer::Chat(Ok(Box::new(completion))); assert_eq!( - answer.record(), + answer.to_record(), AnswerRecord::Chat(Ok( ChatAnswerRecord::new("canned-model").with_reply(Some("canned".to_owned())) )) @@ -278,7 +282,7 @@ fn a_failed_chat_answer_records_the_kinds_tag_and_the_errors_display_text() { "the recorded text keeps the crate-authored specific: {expected}" ); assert!(!expected.contains("hidden"), "the detail is never recorded"); - let record = EffectAnswer::Chat(Err(error)).record(); + let record = EffectAnswer::Chat(Err(error)).to_record(); assert_eq!( record, AnswerRecord::Chat(Err(FailureRecord::new("malformed_response", expected))) @@ -288,7 +292,7 @@ fn a_failed_chat_answer_records_the_kinds_tag_and_the_errors_display_text() { #[test] fn a_tool_call_answer_records_the_outputs_text_and_trust() { - let trusted = EffectAnswer::ToolCall(Ok(ToolOutput::trusted("done"))).record(); + let trusted = EffectAnswer::ToolCall(Ok(ToolOutput::trusted("done"))).to_record(); assert_eq!( trusted, AnswerRecord::ToolCall(Ok(ToolAnswerRecord::new("done", OutputTrust::Trusted))) @@ -299,7 +303,7 @@ fn a_tool_call_answer_records_the_outputs_text_and_trust() { ); assert_eq!(answer_round_trip(&trusted), trusted); - let untrusted = EffectAnswer::ToolCall(Ok(ToolOutput::untrusted(""))).record(); + let untrusted = EffectAnswer::ToolCall(Ok(ToolOutput::untrusted(""))).to_record(); assert_eq!( untrusted, AnswerRecord::ToolCall(Ok(ToolAnswerRecord::new("", OutputTrust::Untrusted))) @@ -311,7 +315,7 @@ fn a_tool_call_answer_records_the_outputs_text_and_trust() { fn a_failed_tool_call_answer_records_the_errors_display_text_and_hides_its_source() { let error = ToolError::new(ToolErrorKind::Backend, "backend failed") .with_source(std::io::Error::other("boom")); - let record = EffectAnswer::ToolCall(Err(error)).record(); + let record = EffectAnswer::ToolCall(Err(error)).to_record(); assert_eq!( record, AnswerRecord::ToolCall(Err(FailureRecord::new("backend", "backend failed"))), @@ -345,7 +349,7 @@ fn a_failure_record_of_each_kind_round_trips_through_json() { ), ]; for (answer, line) in cases { - let record = answer.record(); + let record = answer.to_record(); assert_eq!( serde_json::to_string(&record).expect("a failure record serializes"), line @@ -359,14 +363,14 @@ fn the_store_answer_records_byte_identical_json_for_every_outcome() { // `AnswerRecord::Vfs` holds the `VfsOutcome` itself, and its JSON // is a fixed run-log format: the unit outcome as a bare string, the // payload outcomes in serde's externally tagged form. - let unit = EffectAnswer::Vfs(Ok(VfsOutcome::Unit)).record(); + let unit = EffectAnswer::Vfs(Ok(VfsOutcome::Unit)).to_record(); assert_eq!( serde_json::to_string(&unit).expect("a store answer records"), r#"{"Vfs":{"Ok":"Unit"}}"# ); assert_eq!(answer_round_trip(&unit), unit); - let text = EffectAnswer::Vfs(Ok(VfsOutcome::Text("the read".to_owned()))).record(); + let text = EffectAnswer::Vfs(Ok(VfsOutcome::Text("the read".to_owned()))).to_record(); assert_eq!( serde_json::to_string(&text).expect("a store answer records"), r#"{"Vfs":{"Ok":{"Text":"the read"}}}"# @@ -377,14 +381,14 @@ fn the_store_answer_records_byte_identical_json_for_every_outcome() { "a.txt".to_owned(), "b.txt".to_owned(), ]))) - .record(); + .to_record(); assert_eq!( serde_json::to_string(&paths).expect("a store answer records"), r#"{"Vfs":{"Ok":{"Paths":["a.txt","b.txt"]}}}"# ); assert_eq!(answer_round_trip(&paths), paths); - let flag = EffectAnswer::Vfs(Ok(VfsOutcome::Bool(true))).record(); + let flag = EffectAnswer::Vfs(Ok(VfsOutcome::Bool(true))).to_record(); assert_eq!( serde_json::to_string(&flag).expect("a store answer records"), r#"{"Vfs":{"Ok":{"Bool":true}}}"# @@ -399,7 +403,7 @@ fn a_failed_store_answer_records_the_vfs_errors_tag_and_display_text() { }; let expected = error.to_string(); assert_eq!(expected, "not found: missing.txt"); - let record = EffectAnswer::Vfs(Err(error)).record(); + let record = EffectAnswer::Vfs(Err(error)).to_record(); assert_eq!( record, AnswerRecord::Vfs(Err(FailureRecord::new("not_found", expected))) @@ -410,3 +414,85 @@ fn a_failed_store_answer_records_the_vfs_errors_tag_and_display_text() { ); assert_eq!(answer_round_trip(&record), record); } + +/// Checks that `record`, a public record, is the JSON of `typed`, and that +/// it reads back identical through its text and through the typed form. +#[track_caller] +fn assert_exact_round_trip(record: &Value, typed: &T) +where + T: Serialize + DeserializeOwned + PartialEq + fmt::Debug, +{ + let text = serde_json::to_string(record).expect("a record serializes"); + assert_eq!( + record, + &serde_json::to_value(typed).expect("a typed record converts"), + "the public record is the typed record's JSON: {text}" + ); + let reread: Value = serde_json::from_str(&text).expect("a record's text is JSON"); + assert_eq!(&reread, record, "the text reads back to the same value"); + let back: T = serde_json::from_value(record.clone()).expect("a record reads back typed"); + assert_eq!(&back, typed); + assert_eq!( + &serde_json::to_value(&back).expect("a typed record converts"), + record, + "the typed form converts back to the same value" + ); +} + +#[test] +fn every_kinds_public_record_is_json_that_round_trips_exactly() { + let access = || { + Arc::new( + runvil_vfs::VfsRef::default() + .acquire(runvil_vfs::Origin::new("effect record fixture")) + .expect("the stock backend acquires"), + ) + }; + let effects = [ + chat_effect(three_messages(), Some((RoundId::new(2), 1))), + Effect::Render { + entry: "0.1".to_owned(), + binding: test_binding("writer", "A general model for tests", "test-model"), + offered: tools_pack("Echo the value.", "echo"), + }, + Effect::ToolCall { + tool: ToolId::parse("tools/echo").expect("a fixture tool id"), + alias: "echo".to_owned(), + args: json!({ "value": 0.1, "nested": { "b": 1, "a": [true, null] } }), + origin: ToolCallOrigin::new("run-1", "Only", ToolCaller::Model).with_entry("0.1"), + access: access(), + survives_stop: false, + }, + Effect::Vfs { + access: access(), + op: VfsOp::Write { + path: "note.md".to_owned(), + contents: "hello".to_owned(), + }, + }, + Effect::Timer { + duration: Duration::from_millis(500), + }, + ]; + for effect in &effects { + assert_exact_round_trip(&effect.record(), &effect.to_record()); + } + let answers = [ + EffectAnswer::Chat(Ok(Box::new(finished_completion( + CompletionResult::Text("the reply".to_owned()), + "stop", + )))), + EffectAnswer::Chat(Err(CompletionError::new( + CompletionErrorKind::RateLimited, + "the model backend is limiting the request rate", + ))), + EffectAnswer::Render(tools_pack("Shout the value.", "shout")), + EffectAnswer::ToolCall(Ok(ToolOutput::untrusted(""))), + EffectAnswer::Vfs(Ok(VfsOutcome::Paths(vec!["a.txt".to_owned()]))), + EffectAnswer::Timer, + EffectAnswer::Dropped, + ]; + for answer in &answers { + assert_exact_round_trip(&answer.record(), &answer.to_record()); + } +} diff --git a/crates/runvil-internal/engine/src/execute/run/effect.rs b/crates/runvil-internal/engine/src/execute/run/effect.rs index ed8046e19..a3dde60a8 100644 --- a/crates/runvil-internal/engine/src/execute/run/effect.rs +++ b/crates/runvil-internal/engine/src/execute/run/effect.rs @@ -14,14 +14,14 @@ //! the round. //! //! An [`Effect`] may hold a live handle (a filesystem access) and so does -//! not serialize itself. [`Effect::record`] projects it onto an -//! [`EffectRecord`], the effect minus its handles, which round-trips -//! through serde: a run log stores records, and a later replay compares a -//! re-executed run's records against them. An [`EffectAnswer`] likewise -//! contains values a log cannot hold whole (a completion's bodies, an -//! error's boxed cause); [`EffectAnswer::record`] projects it onto an -//! [`AnswerRecord`], the answer's outcome as a log stores it. The record -//! types sit in the `record` child module and are re-exported here. +//! not serialize itself. [`Effect::record`] projects it onto a JSON value, +//! the effect minus its handles, which round-trips exactly: a run log +//! stores records, and a later replay compares a re-executed run's records +//! against them. An [`EffectAnswer`] likewise contains values a log cannot +//! hold whole (a completion's bodies, an error's boxed cause); +//! [`EffectAnswer::record`] projects it onto the answer's outcome as a log +//! stores it. Both values serialize the crate-private record types in the +//! `record` child module, [`EffectRecord`] and [`AnswerRecord`]. #[path = "effect-record.rs"] mod record; @@ -41,8 +41,9 @@ use crate::lua::{VfsOp, VfsOutcome}; use crate::model::{Completion, CompletionError, Message, ToolSchema}; use crate::model::{ModelBinding, Temperature}; -pub use record::{ - AnswerRecord, ChatAnswerRecord, EffectRecord, FailureRecord, PackToolSets, ToolAnswerRecord, +pub use record::PackToolSets; +pub(crate) use record::{ + AnswerRecord, ChatAnswerRecord, EffectRecord, FailureRecord, ToolAnswerRecord, }; /// The identifier of one in-flight effect, which pairs an issued @@ -230,9 +231,25 @@ fn offered_ids(sets: &PackToolSets) -> PackToolSets { impl Effect { /// Returns the effect's record: the same request with its live handles - /// removed, in a form that a run log stores and a replay compares. + /// removed, as the JSON value that a run log stores and a replay + /// compares. + /// + /// The value is one object keyed by the effect's kind, such as + /// `{"Timer":{"seconds":0.5}}`, and it round-trips exactly through + /// `serde_json`. A `Chat` record holds the round's id, the alias of the + /// model slot it ran under, the frozen invocation settings, the + /// advertised tool names, and the request's messages in wire form after + /// the first `keep`, which repeat round `after`'s request. So a caller + /// that logs records must record every `Chat` effect a step returns, + /// including ones it drops, for each `after` to name a recorded round. #[must_use] - pub fn record(&self) -> EffectRecord { + pub fn record(&self) -> Value { + wire_value(&self.to_record()) + } + + /// Returns the effect's record in the typed form that + /// [`record`](Self::record) serializes. + pub(crate) fn to_record(&self) -> EffectRecord { match self { Effect::Chat { binding, @@ -324,7 +341,7 @@ pub enum EffectAnswer { /// The caller gave up on the effect and dropped it, for example /// because the run was cancelled or the effect's task ended first. If /// the chain still waits on the effect, it resumes with a cancelled - /// error. Uncaught, that error ends the run as `RunResult::Cancelled`. A + /// error. Uncaught, that error ends the run as `RunOutcome::Cancelled`. A /// drop is an answer like any other, so it counts as the effect's one /// answer. Dropped, @@ -332,13 +349,25 @@ pub enum EffectAnswer { impl EffectAnswer { /// Returns the answer's record: the parts of its outcome that a log can - /// hold whole. A failure is recorded as a [`FailureRecord`]: its kind's + /// hold whole, as a JSON value. + /// + /// The value is keyed by the answer's kind, such as + /// `{"ToolCall":{"Ok":{"text":"done","trust":"trusted"}}}` or + /// `"Dropped"`, and it round-trips exactly through `serde_json`. A + /// failure is recorded as `{"kind":..,"message":..}`: its kind's stable /// tag and its display text. A completion is recorded as its reply or - /// its requested tool names, and a render as its rendered tools whole. - /// The round's request and response bodies travel as debug events, and - /// its metrics travel in the turn's event. + /// its requested tool names beside the model and finish reason, and a + /// render as its rendered tools whole. The round's request and response + /// bodies travel as debug events, and its metrics travel in the turn's + /// event. #[must_use] - pub fn record(&self) -> AnswerRecord { + pub fn record(&self) -> Value { + wire_value(&self.to_record()) + } + + /// Returns the answer's record in the typed form that + /// [`record`](Self::record) serializes. + pub(crate) fn to_record(&self) -> AnswerRecord { match self { EffectAnswer::Chat(result) => AnswerRecord::Chat(match result { Ok(completion) => Ok(ChatAnswerRecord::from(completion.as_ref())), diff --git a/crates/runvil-internal/engine/src/execute/run/tests-drops.rs b/crates/runvil-internal/engine/src/execute/run/tests-drops.rs index df6078ebb..6178f3604 100644 --- a/crates/runvil-internal/engine/src/execute/run/tests-drops.rs +++ b/crates/runvil-internal/engine/src/execute/run/tests-drops.rs @@ -6,16 +6,18 @@ //! A `Timer` drop is covered for the timed `tasks.join_any` and //! `tasks.join` waits, and under a run cancel, where the drop wakes no one. +use std::collections::BTreeMap; use std::sync::Arc; +use runvil_types::cancel::CancelHandle; use runvil_types::event::{Event, ReplyOrigin}; use runvil_types::ids::RoundId; -use runvil_types::models::ThinkingMode; +use runvil_types::models::{ModelRole, ThinkingMode}; use runvil_types::tools::{ToolCatalog, ToolId, ToolInfo}; use serde_json::json; -use super::run_context; -use crate::execute::RunResult; +use super::run_config; +use crate::execute::RunOutcome; use crate::execute::run::{ AnswerRecord, Effect, EffectAnswer, EffectId, EffectRecord, Round, Run, Step, }; @@ -24,23 +26,23 @@ use crate::parser::Prompt; use crate::test_support::scenario::{chat_answer, only_effect, test_model_with}; use crate::test_support::scripted_chat::text; -/// A run over one section whose Lua is `body`, with the `writer` role -/// bound to `test-model` and `tools/echo` in the catalog. +/// A run over one section whose Lua is `body`, on the `default` role +/// filled by `test-model`, and `tools/echo` in the catalog, under the +/// cancel flag `cancel`. #[track_caller] -fn bound_run(body: &str) -> Run { - bound_sections(&format!( - "## Only\n\n```lua\nmodels.use('writer')\n{body}\n```\n" - )) +fn bound_run(body: &str, cancel: &CancelHandle) -> Run { + bound_sections( + &format!("## Only\n\n```lua\nmodels.use('default')\n{body}\n```\n"), + cancel, + ) } -/// A run over `sections`, the prompt's H2 sections, with the `writer` role -/// bound to `test-model` and `tools/echo` in the catalog. +/// A run over `sections`, the prompt's H2 sections, on the `default` role +/// filled by `test-model`, and `tools/echo` in the catalog, under the +/// cancel flag `cancel`. #[track_caller] -fn bound_sections(sections: &str) -> Run { - let source = format!( - "---\nname: t\ndescription: d\nrunvil: 0\nmodels:\n writer: {{}}\n\ - ---\n\n# Run\n\n{sections}" - ); +fn bound_sections(sections: &str, cancel: &CancelHandle) -> Run { + let source = format!("---\nname: t\ndescription: d\nrunvil: 0\n---\n\n# Run\n\n{sections}"); let prompt = Prompt::parse(&source, "run-test") .0 .expect("the drop test prompt parses"); @@ -51,13 +53,10 @@ fn bound_sections(sections: &str) -> Run { json!({ "type": "object" }), ); let catalog = ToolCatalog::new([echo]).expect("one tool is a valid catalog"); - let (ctx, requirements) = run_context() + let ctx = run_config() .with_tools(catalog) - .prepare(&prompt, Some(model)); - assert!( - requirements.refusal().is_none(), - "the model fills the prompt's role: {requirements:?}" - ); + .with_models(BTreeMap::from([(ModelRole::Default, model)])) + .with_cancel(cancel.clone()); Run::new(Arc::new(prompt), "", ctx) } @@ -67,19 +66,21 @@ fn bound_sections(sections: &str) -> Run { fn drop_next(run: &mut Run) -> (EffectRecord, AnswerRecord) { let (id, effect) = only_effect(run.step()); let answer = EffectAnswer::Dropped; - let records = (effect.record(), answer.record()); + let records = (effect.to_record(), answer.to_record()); run.resume(id, answer); records } #[test] fn a_dropped_chat_or_tool_call_resumes_a_pcall_with_the_cancelled_error_and_the_run_continues() { + let cancel = CancelHandle::new(); let mut run = bound_run( "local chat_ok, chat_err = pcall(models.infer, 'dropped')\n\ local tool_ok, tool_err = pcall(tools.call, 'tools/echo', { value = 'dropped' })\n\ local kept = models.infer('kept')\n\ return tostring(chat_ok) .. ':' .. chat_err.kind .. '|' \ .. tostring(tool_ok) .. ':' .. tool_err.kind .. '|' .. kept", + &cancel, ); let chat = drop_next(&mut run); assert!( @@ -98,7 +99,7 @@ fn a_dropped_chat_or_tool_call_resumes_a_pcall_with_the_cancelled_error_and_the_ "the record holds the call and its dropped answer: {tool:?}" ); assert!( - !run.cancel_handle().is_cancelled(), + !cancel.is_cancelled(), "the drops left the run's cancel flag clear" ); let (id, effect) = only_effect(run.step()); @@ -109,11 +110,11 @@ fn a_dropped_chat_or_tool_call_resumes_a_pcall_with_the_cancelled_error_and_the_ let completion = Completion::from_result(CompletionResult::Text("kept".to_owned()), "m") .expect("a text result is accepted"); run.resume(id, EffectAnswer::Chat(Ok(Box::new(completion)))); - let Step::Done { result, events } = run.step() else { + let Step::Done { outcome, events } = run.step() else { panic!("the run ends once its last round is answered"); }; - let RunResult::Completed(text) = result else { - panic!("a caught drop leaves the run to succeed: {result:?}"); + let RunOutcome::Completed { final_text: text } = outcome else { + panic!("a caught drop leaves the run to succeed: {outcome:?}"); }; assert_eq!(text, "false:cancelled|false:cancelled|kept"); let turns: Vec = events @@ -132,6 +133,7 @@ fn a_dropped_chat_or_tool_call_resumes_a_pcall_with_the_cancelled_error_and_the_ #[test] fn a_dropped_loop_round_resumes_a_pcall_with_the_cancelled_error_and_the_run_continues() { + let cancel = CancelHandle::new(); let mut run = bound_run( "messages:user('asked')\n\ local loop_ok, loop_err = pcall(models.loop)\n\ @@ -139,6 +141,7 @@ fn a_dropped_loop_round_resumes_a_pcall_with_the_cancelled_error_and_the_run_con models.loop()\n\ return tostring(loop_ok) .. ':' .. loop_err.kind .. '|' .. after_drop \ .. '|' .. #messages .. ':' .. messages[#messages].content", + &cancel, ); let (id, effect) = only_effect(run.step()); assert!( @@ -149,7 +152,7 @@ fn a_dropped_loop_round_resumes_a_pcall_with_the_cancelled_error_and_the_run_con ), "the loop's first round is a chat round numbered 0: {effect:?}" ); - let records = (effect.record(), EffectAnswer::Dropped.record()); + let records = (effect.to_record(), EffectAnswer::Dropped.to_record()); run.resume(id, EffectAnswer::Dropped); assert!( matches!( @@ -159,7 +162,7 @@ fn a_dropped_loop_round_resumes_a_pcall_with_the_cancelled_error_and_the_run_con "the record holds the loop round and its dropped answer: {records:?}" ); assert!( - !run.cancel_handle().is_cancelled(), + !cancel.is_cancelled(), "the drop left the run's cancel flag clear" ); let (id, effect) = only_effect(run.step()); @@ -174,11 +177,11 @@ fn a_dropped_loop_round_resumes_a_pcall_with_the_cancelled_error_and_the_run_con let completion = Completion::from_result(CompletionResult::Text("kept".to_owned()), "m") .expect("a text result is accepted"); run.resume(id, EffectAnswer::Chat(Ok(Box::new(completion)))); - let Step::Done { result, events } = run.step() else { + let Step::Done { outcome, events } = run.step() else { panic!("the run ends once its last round is answered"); }; - let RunResult::Completed(text) = result else { - panic!("a caught drop leaves the run to succeed: {result:?}"); + let RunOutcome::Completed { final_text: text } = outcome else { + panic!("a caught drop leaves the run to succeed: {outcome:?}"); }; assert_eq!( text, "false:cancelled|1|2:kept", @@ -205,26 +208,30 @@ fn an_uncaught_dropped_chat_or_tool_call_ends_the_run_cancelled() { "messages:user('dropped')\nreturn models.loop()", "return tools.call('tools/echo', { value = 'dropped' })", ] { - let mut run = bound_run(body); + let mut run = bound_run(body, &CancelHandle::new()); let _ = drop_next(&mut run); - let Step::Done { result, .. } = run.step() else { + let Step::Done { outcome, .. } = run.step() else { panic!("{body}: the uncaught drop ends the run with nothing outstanding"); }; assert!( - matches!(result, RunResult::Cancelled), - "{body}: the cancelled error ends the run cancelled: {result:?}" + matches!(outcome, RunOutcome::Cancelled), + "{body}: the cancelled error ends the run cancelled: {outcome:?}" ); } } /// A run whose `## Main` runs `main` beside the `## Child` it spawns, whose -/// one round stays out until the test answers it. +/// one round stays out until the test answers it, under the cancel flag +/// `cancel`. #[track_caller] -fn timed_run(main: &str) -> Run { - bound_sections(&format!( - "## Main\n\n```lua\nmodels.use('writer')\n{main}\n```\n\n\ - ## Child\n\n```lua\nmodels.use('writer')\nreturn models.infer('child')\n```\n" - )) +fn timed_run(main: &str, cancel: &CancelHandle) -> Run { + bound_sections( + &format!( + "## Main\n\n```lua\nmodels.use('default')\n{main}\n```\n\n\ + ## Child\n\n```lua\nmodels.use('default')\nreturn models.infer('child')\n```\n" + ), + cancel, + ) } /// A main that catches its timed `wait` over the child, then joins the @@ -264,11 +271,12 @@ fn parked_wait(run: &mut Run) -> (EffectId, EffectId) { #[test] fn a_dropped_timer_raises_the_cancelled_error_into_a_timed_waits_pcall_and_the_run_continues() { for wait in ["tasks.join_any", "tasks.join"] { - let mut run = timed_run(&caught_wait(wait)); + let cancel = CancelHandle::new(); + let mut run = timed_run(&caught_wait(wait), &cancel); let (timer, chat) = parked_wait(&mut run); run.resume(timer, EffectAnswer::Dropped); assert!( - !run.cancel_handle().is_cancelled(), + !cancel.is_cancelled(), "{wait}: the drop left the run's cancel flag clear" ); let Step::Pending { effects, .. } = run.step() else { @@ -279,11 +287,11 @@ fn a_dropped_timer_raises_the_cancelled_error_into_a_timed_waits_pcall_and_the_r "{wait}: the caught drop issued nothing new: {effects:?}" ); run.resume(chat, chat_answer(&text("child"))); - let Step::Done { result, .. } = run.step() else { + let Step::Done { outcome, .. } = run.step() else { panic!("{wait}: the run ends once the child's round is answered"); }; - let RunResult::Completed(out) = result else { - panic!("{wait}: a caught drop leaves the run to succeed: {result:?}"); + let RunOutcome::Completed { final_text: out } = outcome else { + panic!("{wait}: a caught drop leaves the run to succeed: {outcome:?}"); }; assert_eq!( out, "false:cancelled|child", @@ -295,44 +303,52 @@ fn a_dropped_timer_raises_the_cancelled_error_into_a_timed_waits_pcall_and_the_r #[test] fn an_uncaught_dropped_timer_ends_the_run_cancelled() { for wait in ["tasks.join_any", "tasks.join"] { - let mut run = timed_run(&format!( - "local t = tasks.spawn('## Child')\n\ - {wait}({{ t }}, {{ timeout = 30 }})\n\ - return 'waited'" - )); + let mut run = timed_run( + &format!( + "local t = tasks.spawn('## Child')\n\ + {wait}({{ t }}, {{ timeout = 30 }})\n\ + return 'waited'" + ), + &CancelHandle::new(), + ); let (timer, chat) = parked_wait(&mut run); run.resume(timer, EffectAnswer::Dropped); - let Step::Pending { effects, .. } = run.step() else { + let Step::Pending { + effects, withdrawn, .. + } = run.step() + else { panic!("{wait}: the child's round is still out, so the run is not done"); }; assert!(effects.is_empty(), "{wait}: nothing new: {effects:?}"); - assert!( - run.decided(), - "{wait}: the uncaught error decided the run; the child's round is an orphan" + assert_eq!( + withdrawn, + [chat], + "{wait}: the uncaught error decided the run and withdrew the child's round" ); run.resume(chat, EffectAnswer::Dropped); - let Step::Done { result, .. } = run.step() else { + let Step::Done { outcome, .. } = run.step() else { panic!("{wait}: the run ends once the orphan is answered"); }; assert!( - matches!(result, RunResult::Cancelled), - "{wait}: the cancelled error ends the run cancelled: {result:?}" + matches!(outcome, RunOutcome::Cancelled), + "{wait}: the cancelled error ends the run cancelled: {outcome:?}" ); } } #[test] fn under_a_run_cancel_a_dropped_timer_ends_the_run_cancelled_without_resuming_the_wait() { - let mut run = timed_run(&caught_wait("tasks.join_any")); + let cancel = CancelHandle::new(); + let mut run = timed_run(&caught_wait("tasks.join_any"), &cancel); let (timer, chat) = parked_wait(&mut run); - run.cancel(); + cancel.cancel(); run.resume(timer, EffectAnswer::Dropped); run.resume(chat, EffectAnswer::Dropped); - let Step::Done { result, .. } = run.step() else { + let Step::Done { outcome, .. } = run.step() else { panic!("the cancel ends the run once both effects are answered"); }; assert!( - matches!(result, RunResult::Cancelled), - "the wait's pcall never resumed, so the run did not go on: {result:?}" + matches!(outcome, RunOutcome::Cancelled), + "the wait's pcall never resumed, so the run did not go on: {outcome:?}" ); } diff --git a/crates/runvil-internal/engine/src/execute/run/tests-start.rs b/crates/runvil-internal/engine/src/execute/run/tests-start.rs new file mode 100644 index 000000000..00a3fbe94 --- /dev/null +++ b/crates/runvil-internal/engine/src/execute/run/tests-start.rs @@ -0,0 +1,72 @@ +//! The run's first step: a run whose only section returns a literal is +//! done at once, a run that cannot start reports its failure with no run +//! events, and so does a run an answer decides before it starts. + +use super::*; + +#[test] +fn a_run_whose_only_section_returns_a_literal_is_done_at_its_first_step() { + let mut run = run_of("return 'hello'", run_config()); + let Step::Done { + outcome: RunOutcome::Completed { final_text }, + .. + } = run.step() + else { + panic!("the literal run is done at once"); + }; + assert_eq!(final_text, "hello"); +} + +#[test] +fn a_run_that_cannot_start_reports_its_failure_on_the_first_step() { + let source = "---\nname: t\ndescription: d\nrunvil: 7\n---\n\n# Run\n\n## Only\n\ndone\n"; + let prompt = Prompt::parse(source, "run-test") + .0 + .expect("the prompt parses whatever version it declares"); + let mut run = Run::new(Arc::new(prompt), "", run_config()); + let Step::Done { outcome, events } = run.step() else { + panic!("a run that cannot start is done at once"); + }; + assert!(events.is_empty(), "nothing ran, nothing reported"); + assert_eq!( + outcome, + RunOutcome::Failed { + kind: crate::execute::RunErrorKind::Parse, + message: "unsupported runvil version: 7 (this build supports major 0)".to_owned(), + } + ); + assert!( + run.decided_for_test(), + "a run that never started is decided" + ); + let Step::Done { + outcome: RunOutcome::Failed { kind, .. }, + .. + } = run.step() + else { + panic!("a finished run stays done"); + }; + assert_eq!(kind, crate::execute::RunErrorKind::Internal); +} + +#[test] +fn an_answer_before_the_first_step_fails_the_run_with_no_run_events() { + let mut run = run_of("return 'never runs'", run_config()); + run.resume(EffectId(7), EffectAnswer::Timer); + assert!(run.decided_for_test(), "the stray answer decides the run"); + let Step::Done { outcome, events } = run.step() else { + panic!("a run decided before it started is done at its first step"); + }; + assert!( + events.is_empty(), + "a run that never started reports no run events: {events:?}" + ); + let RunOutcome::Failed { kind, message } = outcome else { + panic!("a stray answer fails the run, got {outcome:?}"); + }; + assert_eq!(kind, crate::execute::RunErrorKind::Internal); + assert!( + message.contains("an answer arrived for an effect the run did not issue"), + "{message}" + ); +} diff --git a/crates/runvil-internal/engine/src/execute/run/tests.rs b/crates/runvil-internal/engine/src/execute/run/tests.rs index daa9e21d6..f121dbc7e 100644 --- a/crates/runvil-internal/engine/src/execute/run/tests.rs +++ b/crates/runvil-internal/engine/src/execute/run/tests.rs @@ -8,6 +8,7 @@ use std::sync::Arc; use runvil_model_client::detail::tool_schema_new; use runvil_model_client::model::{ModelInvocation, Temperature}; +use runvil_types::cancel::CancelHandle; use runvil_types::event::{Event, ReplyOrigin}; use runvil_types::ids::RoundId; use runvil_types::tools::ToolId; @@ -21,11 +22,13 @@ use crate::test_support::scenario::{TEST_SEED, only_effect, round_trip, test_bin #[path = "tests-drops.rs"] mod drops; +#[path = "tests-start.rs"] +mod start; -/// A context for the run `run-test` under fixed caller inputs; nothing here +/// A config for the run `run-test` under fixed caller inputs; nothing here /// reads the seed or `sys.when`. -fn run_context() -> RunContext { - RunContext::new( +fn run_config() -> RunConfig { + RunConfig::new( "run-test", TEST_SEED, runvil_types::timestamp::Timestamp::UNIX_EPOCH, @@ -57,7 +60,7 @@ fn a_chat_effect_records_its_round_alias_messages_tools_and_invocation() { origin: ReplyOrigin::Chat, }, }; - let record = effect.record(); + let record = effect.to_record(); assert_eq!( record, EffectRecord::Chat { @@ -128,7 +131,7 @@ fn a_tool_call_record_keeps_the_identity_alias_args_and_origin_but_no_access_or_ access, survives_stop: true, }; - let record = effect.record(); + let record = effect.to_record(); assert_eq!( record, EffectRecord::ToolCall { @@ -177,7 +180,7 @@ fn a_store_effect_records_its_operation_and_drops_the_access_handle() { end: None, }, }; - let record = effect.record(); + let record = effect.to_record(); assert_eq!( record, EffectRecord::Vfs { @@ -202,7 +205,7 @@ fn a_timer_effect_records_its_duration_in_seconds() { let effect = Effect::Timer { duration: std::time::Duration::from_millis(250), }; - let record = effect.record(); + let record = effect.to_record(); assert_eq!(record, EffectRecord::Timer { seconds: 0.25 }); assert_eq!(round_trip(&record), record); } @@ -219,7 +222,7 @@ fn prompt_of(body: &str) -> Arc { } /// A run of [`prompt_of`]`(body)`. -fn run_of(body: &str, config: RunContext) -> Run { +fn run_of(body: &str, config: RunConfig) -> Run { Run::new(prompt_of(body), "", config) } @@ -242,9 +245,10 @@ fn a_run_is_send() { #[test] fn done_is_withheld_while_a_store_effect_is_outstanding_and_delivered_after_dropped() { + let cancel = CancelHandle::new(); let mut run = run_of( "store.write('notes.md', 'kept')\nreturn 'unreachable'", - run_context(), + run_config().with_cancel(cancel.clone()), ); let (id, effect) = only_effect(run.step()); assert!( @@ -254,7 +258,7 @@ fn done_is_withheld_while_a_store_effect_is_outstanding_and_delivered_after_drop // The run is cancelled while the store operation is out. The next step // tears the run down and reports its end, but the effect still owes // its answer, so `Done` waits. - run.cancel(); + cancel.cancel(); let step = run.step(); let Step::Pending { effects, events, .. @@ -269,39 +273,42 @@ fn done_is_withheld_while_a_store_effect_is_outstanding_and_delivered_after_drop ); run.resume(id, EffectAnswer::Dropped); let step = run.step(); - let Step::Done { result, .. } = step else { + let Step::Done { outcome, .. } = step else { panic!("Done follows the last answer, got {step:?}"); }; - assert!( - matches!(result, RunResult::Cancelled), - "the cancelled run reports as cancelled: {result:?}" + assert_eq!( + outcome, + RunOutcome::Cancelled, + "the cancelled run reports as cancelled" ); } #[test] fn a_dropped_answer_resumes_a_waiting_chain_with_the_cancelled_error() { - let mut run = run_of("return store.exists('x')", run_context()); + let mut run = run_of("return store.exists('x')", run_config()); let (id, effect) = only_effect(run.step()); assert!(matches!(effect, Effect::Vfs { .. })); run.resume(id, EffectAnswer::Dropped); let step = run.step(); - let Step::Done { result, .. } = step else { + let Step::Done { outcome, .. } = step else { panic!("the dropped store operation ends the run, got {step:?}"); }; - assert!( - matches!(result, RunResult::Cancelled), - "the chain resumed with the cancelled error and the run reports it: {result:?}" + assert_eq!( + outcome, + RunOutcome::Cancelled, + "the chain resumed with the cancelled error and the run reports it" ); } #[test] fn an_orphaned_effects_real_answer_is_discarded_and_still_counts_as_the_answer() { + let cancel = CancelHandle::new(); let mut run = run_of( "store.write('notes.md', 'kept')\nreturn 'unreachable'", - run_context(), + run_config().with_cancel(cancel.clone()), ); let (id, _) = only_effect(run.step()); - run.cancel(); + cancel.cancel(); assert!(matches!(run.step(), Step::Pending { .. })); // The caller performed the operation before it learned of the cancel: // its answer is the effect's one answer, discarded rather than applied. @@ -310,7 +317,7 @@ fn an_orphaned_effects_real_answer_is_discarded_and_still_counts_as_the_answer() matches!( run.step(), Step::Done { - result: RunResult::Cancelled, + outcome: RunOutcome::Cancelled, .. } ), @@ -320,7 +327,7 @@ fn an_orphaned_effects_real_answer_is_discarded_and_still_counts_as_the_answer() #[test] fn an_answer_for_an_unissued_effect_is_an_internal_error() { - let mut run = run_of("return store.exists('x')", run_context()); + let mut run = run_of("return store.exists('x')", run_config()); let (id, _) = only_effect(run.step()); run.resume(EffectId(id.0 + 99), EffectAnswer::Timer); // The unknown id ended the run; the real effect is now an orphan whose @@ -330,33 +337,33 @@ fn an_answer_for_an_unissued_effect_is_an_internal_error() { }; assert!(ended(&events)); run.resume(id, EffectAnswer::Dropped); - let Step::Done { result, .. } = run.step() else { + let Step::Done { outcome, .. } = run.step() else { panic!("Done follows the orphan's answer"); }; - let RunResult::Failure(error) = result else { - panic!("an unknown id fails the run, got {result:?}"); + let RunOutcome::Failed { kind, message } = outcome else { + panic!("an unknown id fails the run, got {outcome:?}"); }; - assert_eq!(error.kind(), crate::execute::RunErrorKind::Internal); + assert_eq!(kind, crate::execute::RunErrorKind::Internal); assert!( - error.to_string().contains("did not issue"), - "the failure names the unknown id: {error}" + message.contains("did not issue"), + "the failure names the unknown id: {message}" ); } #[test] fn an_answer_of_the_wrong_kind_for_a_pending_effect_fails_loudly() { - let mut run = run_of("return store.exists('x')", run_context()); + let mut run = run_of("return store.exists('x')", run_config()); let (id, _) = only_effect(run.step()); run.resume(id, EffectAnswer::Timer); - let Step::Done { result, .. } = run.step() else { + let Step::Done { outcome, .. } = run.step() else { panic!("the mismatch ends the run with nothing outstanding"); }; - let RunResult::Failure(error) = result else { - panic!("a wrong-kind answer fails the run, got {result:?}"); + let RunOutcome::Failed { message, .. } = outcome else { + panic!("a wrong-kind answer fails the run, got {outcome:?}"); }; assert!( - error.to_string().contains("effect's own kind"), - "the mismatch is a loud invariant failure: {error}" + message.contains("effect's own kind"), + "the mismatch is a loud invariant failure: {message}" ); } @@ -370,7 +377,7 @@ fn a_child_cancel_handles_cancel_is_observed_by_the_instruction_hook() { "local n = 0\nwhile true do\n n = n + 1\n if n == 1 then log('looping') end\nend", ), "", - run_context().with_cancel(child), + run_config().with_cancel(child), |fresh| { state = Some(fresh.clone()); fresh @@ -396,94 +403,73 @@ fn a_child_cancel_handles_cancel_is_observed_by_the_instruction_hook() { }); let step = run.step(); canceller.join().expect("the canceller thread finishes"); - let Step::Done { result, events } = step else { + let Step::Done { outcome, events } = step else { panic!("the hook aborts the loop and nothing is outstanding, got {step:?}"); }; - assert!(matches!(result, RunResult::Cancelled), "got {result:?}"); + assert_eq!(outcome, RunOutcome::Cancelled); assert!(ended(&events), "the end boundary is on the final step"); } #[test] -fn a_context_without_a_caller_handle_shares_its_one_flag_with_prepare_and_the_run() { - let source = "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n# Run\n\n## Only\n\n```lua\nreturn 'x'\n```\n"; - let prompt = Prompt::parse(source, "run-test") - .0 - .expect("the run test prompt parses"); - // The flag `prepare` hands the Packs is the context's own. - let (ctx, _) = run_context().prepare(&prompt, None); - let packs_flag = ctx.cancel.clone(); - let mut run = Run::new(Arc::new(prompt), "", ctx); - assert!(!packs_flag.is_cancelled()); - assert!(!run.cancel_handle().is_cancelled()); - run.cancel(); - assert!( - packs_flag.is_cancelled(), - "the run's cancel sets the flag the Packs hold" - ); - assert!( - run.cancel_handle().is_cancelled(), - "the run's own handle is the same flag" +fn a_config_without_a_caller_handle_shares_its_one_flag_with_the_run() { + let ctx = run_config(); + let config_flag = ctx.cancel.clone(); + let mut run = run_of("return 'x'", ctx); + config_flag.cancel(); + let Step::Done { outcome, .. } = run.step() else { + panic!("the cancelled run has nothing outstanding"); + }; + assert_eq!( + outcome, + RunOutcome::Cancelled, + "the run polls the flag the config minted" ); } #[test] -fn a_run_is_decided_once_its_end_is_reported_while_done_is_withheld() { +fn the_step_that_decides_the_run_withdraws_its_outstanding_effect_while_done_is_withheld() { + let cancel = CancelHandle::new(); let mut run = run_of( "store.write('notes.md', 'kept')\nreturn 'unreachable'", - run_context(), - ); - assert!(!run.decided(), "a fresh run is undecided"); - let (id, _) = only_effect(run.step()); - assert!(!run.decided(), "a run waiting on an answer is undecided"); - run.cancel(); - assert!( - matches!(run.step(), Step::Pending { .. }), - "Done waits on the store effect" + run_config().with_cancel(cancel.clone()), ); + assert!(!run.decided_for_test(), "a fresh run is undecided"); + let Step::Pending { + effects, withdrawn, .. + } = run.step() + else { + panic!("the run parks on its store effect"); + }; + let id = only_effect_id(&effects); + assert!(withdrawn.is_empty(), "a live run withdraws nothing"); assert!( - run.decided(), - "the run is decided before Done, so the caller can drop what it holds" + !run.decided_for_test(), + "a run waiting on an answer is undecided" ); - run.resume(id, EffectAnswer::Dropped); - assert!(matches!(run.step(), Step::Done { .. })); - assert!(run.decided(), "a finished run stays decided"); -} - -#[test] -fn a_run_whose_only_section_returns_a_literal_is_done_at_its_first_step() { - let mut run = run_of("return 'hello'", run_context()); - let Step::Done { - result: RunResult::Completed(text), - .. + cancel.cancel(); + let Step::Pending { + effects, withdrawn, .. } = run.step() else { - panic!("the literal run is done at once"); + panic!("Done waits on the store effect"); }; - assert_eq!(text, "hello"); + assert!(effects.is_empty(), "a torn-down run issues nothing"); + assert_eq!( + withdrawn, + [id], + "the deciding step withdraws the store effect, so the caller drops it" + ); + assert!(run.decided_for_test(), "the run is decided before Done"); + run.resume(id, EffectAnswer::Dropped); + assert!(matches!(run.step(), Step::Done { .. })); + assert!(run.decided_for_test(), "a finished run stays decided"); } -#[test] -fn a_stillborn_run_reports_its_failure_on_the_first_step() { - let source = "---\nname: t\ndescription: d\nrunvil: 7\n---\n\n# Run\n\n## Only\n\ndone\n"; - let prompt = Prompt::parse(source, "run-test") - .0 - .expect("the prompt parses whatever version it declares"); - let mut run = Run::new(Arc::new(prompt), "", run_context()); - let Step::Done { result, events } = run.step() else { - panic!("a run that cannot start is done at once"); - }; - assert!(events.is_empty(), "nothing ran, nothing reported"); - let RunResult::Failure(error) = result else { - panic!("an unsupported version fails, got {result:?}"); - }; - assert_eq!(error.kind(), crate::execute::RunErrorKind::Version); - assert!(run.decided(), "a run that never started is decided"); - let Step::Done { - result: RunResult::Failure(again), - .. - } = run.step() - else { - panic!("a spent run stays done"); +/// The id of the one effect in `effects`. +#[track_caller] +fn only_effect_id(effects: &[IssuedEffect]) -> EffectId { + let [issued] = effects else { + panic!("exactly one effect is issued: {effects:?}"); }; - assert_eq!(again.kind(), crate::execute::RunErrorKind::Internal); + issued.id } diff --git a/crates/runvil-internal/engine/src/execute/scheduler.rs b/crates/runvil-internal/engine/src/execute/scheduler.rs index ae434c166..193a62ea5 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler.rs @@ -26,10 +26,10 @@ //! `Error::Internal` from a scheduler method marks a state the scheduler's //! own invariants rule out: a scheduler or shim bug. //! -//! [`RunState`] stays the ambient shared read-mostly context, cloned into -//! chains and callbacks; the scheduler owns the run's copy and is the -//! exclusively owned mutable counterpart. The two are deliberately not -//! merged: `RunState` is cloned into callbacks, while the scheduler must +//! [`ChainState`] stays the ambient shared read-mostly context, cloned into +//! chains and callbacks; the scheduler owns the root chain's copy and is +//! the exclusively owned mutable counterpart. The two are deliberately not +//! merged: `ChainState` is cloned into callbacks, while the scheduler must //! stay unreachable from the callback layer. The prompt tree is shared //! through the context's `Arc`, so a chain names its position in it by //! path ([`SectionPos`](crate::heading_address::SectionPos)) rather than by @@ -90,7 +90,7 @@ mod walk; use std::collections::VecDeque; use std::sync::Arc; -use runvil_types::event::lifecycle; +use runvil_types::cancel::CancelHandle; use runvil_vfs::Origin; use runvil_vfs::detail::ScopeHandle; @@ -98,7 +98,7 @@ use crate::lua::Answer; use crate::parser::{Block, Prompt}; use crate::{Error, Result}; -use super::context::RunState; +use super::context::ChainState; use super::run::EffectAnswer; use admission::Admission; use chain_ops::ChainOps; @@ -136,11 +136,13 @@ fn prompt_origin(prompt: &Prompt, label: &str, blocks: &[Block]) -> Origin { /// The run's phase, as the step loop reads it. enum Phase { - /// No chain has started: the next step starts the H1 pass or the walk. + /// No chain has started: the next step checks that the run can start, + /// then starts the H1 pass or the walk. Fresh, /// Chains run; the outcome is undecided. Running, - /// The outcome is decided and the run's end boundary reported; `Done` + /// The outcome is decided and the run's end boundary reported, unless + /// the run ended before it started and so reported nothing; `Done` /// waits on the outstanding effects. Ending(Result), /// `Done` was returned. @@ -150,9 +152,10 @@ enum Phase { /// The coroutine protocol's driver: the chain arena, ready queue, effect /// table, task arena, and admission, owned outright by the run. pub(crate) struct Scheduler { - /// The ambient run context, cloned into chains and forked by call - /// chains. - ctx: RunState, + /// The root chain's state: cloned to start the H1 pass and the walk, + /// forked at the handoff between them, and read for the root task's + /// emitter, which reports the run's end. + ctx: ChainState, /// The chain arena: append-only, indexed by [`ChainIndex`]. chains: ChainArena, /// Chains eligible to resume (FIFO); empty whenever a step returns, @@ -177,6 +180,9 @@ pub(crate) struct Scheduler { /// root-start sites (`h1::start_h1` and `walk::start_root_walk`) are its /// only writers, each at the acquisition that opens the run's identity. scope: Option, + /// The error behind the failed outcome the last `Done` carried. + #[cfg(test)] + failure: Option, } impl Drop for Scheduler { @@ -186,14 +192,9 @@ impl Drop for Scheduler { } impl Scheduler { - /// Builds the scheduler for one run over `ctx`'s prompt and reports - /// the run's start, so the first step's events open with it. - pub(super) fn new(ctx: RunState) -> Self { - // The run's boundaries are events like every other report: pushed - // into the buffer under the root task, so the caller sees them in - // order with the sections between them. - ctx.emitter() - .report(ctx.prompt().title(), lifecycle::RUN_STARTED); + /// Builds the scheduler for one run over `ctx`'s prompt. Nothing runs + /// or reports until the first step. + pub(super) fn new(ctx: ChainState) -> Self { Self { ctx, chains: ChainArena::new(), @@ -203,6 +204,8 @@ impl Scheduler { admission: Admission::default(), phase: Phase::Fresh, scope: None, + #[cfg(test)] + failure: None, } } @@ -221,11 +224,28 @@ impl Scheduler { } /// Whether the run's outcome is decided: its end boundary is reported, - /// so at most the orphans' answers stand between the run and `Done`. + /// or the run ended before it started, so at most the orphans' answers + /// stand between the run and `Done`. + #[cfg(test)] pub(super) fn decided(&self) -> bool { matches!(self.phase, Phase::Ending(_) | Phase::Done) } + /// The run's cancel flag. + pub(super) fn cancel_handle(&self) -> &CancelHandle { + self.ctx.cancel() + } + + /// The name of the run's phase, for the run's `Debug` output. + pub(super) fn phase_name(&self) -> &'static str { + match self.phase { + Phase::Fresh => "fresh", + Phase::Running => "running", + Phase::Ending(_) => "ending", + Phase::Done => "done", + } + } + /// Answers `id` at once with `answer` and queues it to resume: the /// inline-answer path of every request the core keeps, and of a chain /// end answering another chain. diff --git a/crates/runvil-internal/engine/src/execute/scheduler/chain.rs b/crates/runvil-internal/engine/src/execute/scheduler/chain.rs index 748447940..ae9d009ae 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/chain.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/chain.rs @@ -5,16 +5,17 @@ //! transitively blocks on or owns. use std::num::NonZeroUsize; -use std::sync::Arc; +use std::sync::{Arc, Mutex}; use runvil_types::ids::{AbandonReason, ChainId, TaskId}; use runvil_vfs::Access; -use crate::execute::context::RunState; +use crate::execute::context::ChainState; use crate::execute::run::EffectId; use crate::execute::section_context::TaskSeed; use crate::heading_address::SectionPos; use crate::lua::Answer; +use crate::model::ModelBinding; use crate::{Error, Result}; use super::chain_record::{CallDepth, ChainKind, Walk}; @@ -31,6 +32,21 @@ pub(super) enum ChainStep { RunEnded(Result), } +impl Chain { + /// Returns the chain's current model: the binding `models.use` last + /// stored, else the model the chain started on. `None` means the caller + /// supplied no model. + /// + /// # Errors + /// Returns [`Error::Internal`] when the selection's lock is poisoned. + pub(super) fn current_model(&self) -> Result> { + self.selection + .lock() + .map(|selection| selection.clone()) + .map_err(|_| Error::internal("model selection lock was poisoned")) + } +} + /// Everything a new chain starts from. pub(super) struct ChainStart { /// The chain's hierarchical id. @@ -39,7 +55,13 @@ pub(super) struct ChainStart { /// the H1 pass's for the root walk that follows it. pub(super) counters: Counters, /// The chain's fork of the run state. - pub(super) ctx: RunState, + pub(super) ctx: ChainState, + /// The model the chain's current model starts on: the `default` binding + /// for the H1 pass and a root walk with no pass before it, the H1 + /// pass's ending model for the walk that follows it, and a copy of the + /// parent's current model, taken when the call or spawn was dispatched, + /// for a `call` child or a task. + pub(super) model: Option, /// The chain's VFS access capability: the root identity for the H1 /// pass and the walk, the caller's for a `call` child, a fork of the /// spawner's for a task. @@ -86,6 +108,7 @@ impl Scheduler { seed: TaskSeed::default(), parked: None, ctx: start.ctx, + selection: Arc::new(Mutex::new(start.model)), access: Some(start.access), entered: None, walk: Walk::new(start.start), @@ -111,13 +134,14 @@ impl Scheduler { /// `u32`, leaving the ready queue unchanged. #[expect( clippy::too_many_arguments, - reason = "a root start inlines the `ChainStart` it would build, so no caller can set `lineage` or `call_depth`, and the four values it forwards verbatim from its site stay explicit" + reason = "a root start inlines the `ChainStart` it would build, so no caller can set `lineage` or `call_depth`, and the five values it forwards verbatim from its site stay explicit" )] pub(super) fn start_root_chain( &mut self, kind: ChainKind, counters: Counters, - ctx: RunState, + ctx: ChainState, + model: Option, access: Arc, start: SectionPos, var: serde_json::Value, @@ -127,6 +151,7 @@ impl Scheduler { lineage: ChainId::root(), counters, ctx, + model, access, start, kind, diff --git a/crates/runvil-internal/engine/src/execute/scheduler/chain_ops.rs b/crates/runvil-internal/engine/src/execute/scheduler/chain_ops.rs index 3abe338df..8470efefb 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/chain_ops.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/chain_ops.rs @@ -64,7 +64,7 @@ impl Scheduler { impl ChainOps<'_> { /// The prompt's shared handle: the chain resolves its walk position /// against this clone so the tree borrow never pins the chain. The - /// chain's own context fork is the source, which is the one `RunState` + /// chain's own context fork is the source, which is the one `ChainState` /// the chain reads the prompt from. pub(super) fn prompt(&self) -> Arc { Arc::clone(self.chain.ctx.prompt()) diff --git a/crates/runvil-internal/engine/src/execute/scheduler/chain_record.rs b/crates/runvil-internal/engine/src/execute/scheduler/chain_record.rs index dbcce33d6..cc6a42b9e 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/chain_record.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/chain_record.rs @@ -5,14 +5,14 @@ use std::fmt; use std::num::NonZeroUsize; -use std::sync::Arc; +use std::sync::{Arc, Mutex}; use mlua::Thread; use runvil_types::ids::{ChainId, TaskId}; use runvil_vfs::Access; use crate::error::DepthVia; -use crate::execute::context::RunState; +use crate::execute::context::ChainState; use crate::execute::section_context::{SectionContext, TaskSeed}; use crate::execute::walk_target::JumpTarget; use crate::heading_address::SectionPos; @@ -285,7 +285,10 @@ pub(super) struct Chain { /// through `with_args` when the call or spawn gave an input. A task's /// fork also reports under its own task id with its own turn counter /// (`with_task`). - pub(super) ctx: RunState, + pub(super) ctx: ChainState, + /// The chain's current model, shared with each section VM it enters: + /// it starts on the chain's starting model; `models.use` rewrites it. + pub(super) selection: Arc>>, /// The chain's VFS access capability, installed into each section VM /// the chain enters: the walk and the H1 pass share one root /// identity, a call chain borrows its parent's (a blocking child is diff --git a/crates/runvil-internal/engine/src/execute/scheduler/chain_record/arena.rs b/crates/runvil-internal/engine/src/execute/scheduler/chain_record/arena.rs index 45b8315f1..52b5434f2 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/chain_record/arena.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/chain_record/arena.rs @@ -45,13 +45,13 @@ impl Chain { kind: super::ChainKind, concurrency: std::num::NonZeroUsize, ) -> Chain { - use std::sync::Arc; + use std::sync::{Arc, Mutex}; use runvil_types::ids::TaskId; use super::{CallDepth, ChatAnchor, Counters, Walk}; - use crate::execute::RunContext; - use crate::execute::context::RunState; + use crate::execute::RunConfig; + use crate::execute::context::ChainState; use crate::execute::section_context::TaskSeed; use crate::heading_address::{SectionIndex, SectionPos}; use crate::parser::Prompt; @@ -64,10 +64,10 @@ impl Chain { let prompt = Prompt::parse(source, "chain-test") .0 .expect("the test prompt parses"); - let ctx = RunState::new( + let ctx = ChainState::root( Arc::new(prompt), "", - RunContext::new("chain-test", TEST_SEED, TEST_STARTED_AT), + RunConfig::new("chain-test", TEST_SEED, TEST_STARTED_AT), ); Chain { task: TaskId::from(lineage.clone()), @@ -77,6 +77,7 @@ impl Chain { seed: TaskSeed::default(), parked: None, ctx, + selection: Arc::new(Mutex::new(None)), access: None, entered: None, walk: Walk::new(SectionPos::top(SectionIndex::FIRST)), diff --git a/crates/runvil-internal/engine/src/execute/scheduler/chat.rs b/crates/runvil-internal/engine/src/execute/scheduler/chat.rs index 6efe6cd2d..e7e9b3bde 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/chat.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/chat.rs @@ -36,7 +36,7 @@ use crate::execute::scope::{SectionOffers, freeze_tool_sets, offered_packs, tool use crate::execute::turn::{Served, advance_turn, report_model_turn}; use crate::lua::{ Answer, ChatOutcome, ChatResult, MessageList, OverflowReason, RenderedTools, ToolSet, - output_reserve, precheck, project_messages, resolve_model_binding, wire_name_of, + output_reserve, precheck, project_messages, wire_name_of, }; use crate::model::{ Completion, CompletionErrorKind, CompletionResult, ModelBinding, ToolCall, detail, @@ -108,7 +108,7 @@ impl ChainOps<'_> { let binding = if let Some(binding) = binding { binding } else { - let current = resolve_model_binding(chain.ctx.models(), vm.model_selection())?; + let current = vm.current_model()?; current.ok_or_else(|| Error::ModelRequired { section: section.clone(), })? diff --git a/crates/runvil-internal/engine/src/execute/scheduler/dispatch.rs b/crates/runvil-internal/engine/src/execute/scheduler/dispatch.rs index 85a7871a0..0db98e142 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/dispatch.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/dispatch.rs @@ -21,7 +21,7 @@ use runvil_types::event::ReplyOrigin; use crate::error::DepthVia; use crate::execute::run::Effect; use crate::execute::section_context::TaskSeed; -use crate::lua::{Answer, Request, VfsOp, resolve_model_binding}; +use crate::lua::{Answer, Request, VfsOp}; use crate::model::{Message, ModelBinding}; use crate::{Error, Result}; @@ -167,6 +167,9 @@ impl Scheduler { // A call chain's effective limit starts with its caller's: the // tasks it spawns run within the caller's share. let child_concurrency = chain.concurrency; + // The child starts on a copy of the caller's current model, taken + // now, at dispatch. + let model = chain.current_model()?; // `chain`'s arena borrow ends here; the resolution names the // target's slice by path, so nothing borrows the arena across it. let resolved = self.with_chain(id, |ops| ops.resolve_chain_target(target))?; @@ -176,6 +179,7 @@ impl Scheduler { lineage, counters: Counters::default(), ctx: child_ctx, + model, access, start: resolved.pos, kind: ChainKind::Call { parent: id }, @@ -206,7 +210,10 @@ impl ChainOps<'_> { let binding = if let Some(binding) = binding { binding } else { - resolve_model_binding(chain.ctx.models(), chain.frame()?.vm().model_selection())? + chain + .frame()? + .vm() + .current_model()? .ok_or_else(|| Error::ModelRequired { section: chain.section_name().to_owned(), })? diff --git a/crates/runvil-internal/engine/src/execute/scheduler/drive.rs b/crates/runvil-internal/engine/src/execute/scheduler/drive.rs index 1d14b32ee..16082bff2 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/drive.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/drive.rs @@ -1,8 +1,8 @@ //! The run-level step: `resume -> match request -> dispatch -> resume with //! answer`, run until no chain can proceed without an answer from the -//! caller. The step starts the run's first chain on its first call, drains -//! the ready queue, and returns the effects the drain issued with the -//! events it reported. +//! caller. The step checks that the run can start and starts its first +//! chain on its first call, drains the ready queue, and returns the +//! effects the drain issued with the events it reported. //! An empty ready queue with no parked effect and no queued chain that can //! be admitted is a stall, which fails loudly rather than hangs. //! The run's `Done` is withheld until @@ -11,10 +11,11 @@ use runvil_types::event::lifecycle; use runvil_types::ids::AbandonReason; +use runvil_vfs::Origin; -use crate::execute::RunResult; -use crate::execute::error::RunError; +use crate::execute::RunOutcome; use crate::execute::run::{EffectAnswer, EffectId, Step}; +use crate::parser::ParseErrorKind; use crate::{Error, Result}; use super::chain::ChainStep; @@ -25,13 +26,31 @@ impl Scheduler { /// Drains the ready queue and returns the step's outcome: the effects /// issued and the events reported while chains ran, or the run's /// result once it is over and every issued effect is answered. The - /// first call starts the H1 pass when the prompt has H1 blocks, else - /// the root chain over the prompt's sections. + /// first call checks that the run can start, then starts the H1 pass + /// when the prompt has H1 blocks, else the root chain over the prompt's + /// sections. A run that cannot start ends with that failure, or as + /// cancelled when its cancel flag is already set, and reports no run + /// events. pub(crate) fn step(&mut self) -> Step { if matches!(self.phase, Phase::Fresh) { - self.phase = Phase::Running; - if let Err(error) = self.start() { - self.end(Err(error)); + if let Err(error) = self.check_start() { + let error = if self.ctx.cancel().is_cancelled() { + Error::Interrupted + } else { + error + }; + self.phase = Phase::Ending(Err(error)); + } else { + self.phase = Phase::Running; + // The run's boundaries are events like every other report: + // pushed into the buffer under the root task, so the caller + // sees them in order with the sections between them. + self.ctx + .emitter() + .report(self.ctx.prompt().title(), lifecycle::RUN_STARTED); + if let Err(error) = self.start() { + self.end(Err(error)); + } } } if matches!(self.phase, Phase::Running) { @@ -67,16 +86,12 @@ impl Scheduler { // orphan's store effect still runs in the live scope. self.end_scope(); Step::Done { - result: match result { - Ok(text) => RunResult::Completed(text), - Err(Error::Interrupted) => RunResult::Cancelled, - Err(error) => RunResult::Failure(RunError::from(error)), - }, + outcome: self.outcome(result), events, } } Phase::Done => Step::Done { - result: RunResult::Failure(RunError::from(Error::internal( + outcome: self.outcome(Err(Error::internal( "a finished run cannot be stepped again", ))), events, @@ -92,6 +107,30 @@ impl Scheduler { } } + /// The outcome the run ends with for `result`: an interruption that + /// reached the end is a cancellation, and any other error a failure. + #[cfg_attr( + not(test), + expect( + clippy::unused_self, + reason = "only a test build keeps the failure's error on the scheduler" + ) + )] + fn outcome(&mut self, result: Result) -> RunOutcome { + match result { + Ok(final_text) => RunOutcome::Completed { final_text }, + Err(Error::Interrupted) => RunOutcome::Cancelled, + Err(error) => { + let outcome = error.to_outcome(); + #[cfg(test)] + { + self.failure = Some(error); + } + outcome + } + } + } + /// Applies one effect's answer. An answer for an orphaned effect (its /// chain stopped waiting) is discarded; an answer for an id the run /// never issued, or a second answer for one effect, ends the run with @@ -140,6 +179,48 @@ impl Scheduler { Ok(()) } + /// Checks that the run can start: the version gate, then the declared + /// store, then what the prompt requires of the config. + /// + /// # Errors + /// Returns [`Error::UnsupportedVersion`] or a structural parse error for + /// a prompt that is not a supported runvil prompt, [`Error::Store`] + /// when the handle declares no store or the store's backend fails the + /// probe, or [`Error::RequirementsUnmet`] with the refusal notice when + /// the config lacks a declared Pack or a model the prompt's roles + /// require, or names a Pack id the Engine reserves. + fn check_start(&self) -> Result<()> { + let frontmatter = self.ctx.prompt().frontmatter(); + match frontmatter.runvil() { + Some(0) => {} + Some(other) => return Err(Error::UnsupportedVersion(other)), + None => { + return Err(Error::parse( + ParseErrorKind::Structure, + "not a runvil prompt: no runvil version", + ) + .with_prompt_name(frontmatter.name())); + } + } + // Every run and every store call needs a declared store: the probe + // stats the store root through a throwaway capability, so a handle + // with no declaration fails here rather than on the first store call, + // and a declared-but-failing backend's error fails the run. + let probe = self + .ctx + .vfs() + .acquire(Origin::new("store mount probe")) + .map_err(Error::store)?; + runvil_vfs::detail::probe_store(&probe).map_err(Error::store)?; + if let Some(notice) = self.ctx.refusal() { + return Err(Error::RequirementsUnmet { + notice: notice.to_owned(), + cause: None, + }); + } + Ok(()) + } + /// Runs every ready chain to its next suspension point. Cancellation /// is polled before each chain step: the instruction hook covers /// running Lua, and a cancel that arrives while every chain is @@ -183,19 +264,23 @@ impl Scheduler { /// well has none left, its root chain having settled its own), tears /// every chain down (the suspended chains' frames /// drop unarmed - no `SECTION_FINISHED` - and every effect still out - /// with the caller becomes an orphan the caller still answers), reports - /// the run's end boundary after every task terminal, and holds - /// `result` until the orphans are answered. A second decision keeps - /// the first: the outcome that ended the run is the record. + /// with the caller becomes an orphan the caller still answers, listed + /// as withdrawn), reports the run's end boundary after every task + /// terminal, and holds `result` until the orphans are answered. A + /// second decision keeps the first: the outcome that ended the run is + /// the record. A run that has not started ends like one that could not + /// start: it holds `result` and reports nothing. fn end(&mut self, result: Result) { - if matches!(self.phase, Phase::Ending(_) | Phase::Done) { - return; + match self.phase { + Phase::Ending(_) | Phase::Done => return, + Phase::Fresh => { + self.phase = Phase::Ending(result); + return; + } + Phase::Running => {} } self.settle_all_tasks(AbandonReason::RunTerminated); self.teardown(); - // Settling withdrew the abandoned tasks' calls, but teardown orphans - // the rest unlisted, so a decided run lists none rather than some. - self.effects.clear_withdrawn(); self.ctx.emitter().report( self.ctx.prompt().title(), if result.is_ok() { @@ -208,9 +293,10 @@ impl Scheduler { } /// Drops every chain's live state in the teardown order (the suspended - /// coroutine, then the frame unarmed, then the access capability) and - /// orphans every awaited effect. The task entries keep their terminal - /// state for inspection; every entry is terminal by now, `end` having + /// coroutine, then the frame unarmed, then the access capability), + /// orphans every awaited effect, and withdraws every outstanding + /// effect. The task entries keep their terminal state for inspection; + /// every entry is terminal by now, `end` having /// settled the live ones, so nothing here reports. Each chain joins /// the tasks it owns before its access drops, closing the run's /// happens-before record. diff --git a/crates/runvil-internal/engine/src/execute/scheduler/effects-tests.rs b/crates/runvil-internal/engine/src/execute/scheduler/effects-tests.rs index dec1bbd3a..83fe88816 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/effects-tests.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/effects-tests.rs @@ -1,8 +1,9 @@ //! Tests for the effect table alone, one rule each: an issued effect takes //! exactly one answer, an orphan's answer is discarded, unknown ids and -//! second answers are reported, only orphaned tool calls are withdrawn, in -//! id order, and an effect orphaned in its own step still goes out with -//! that step. +//! second answers are reported, only orphaned tool calls are withdrawn +//! until teardown withdraws every outstanding effect, in id order, a step +//! lists only withdrawals still owed an answer, and an effect orphaned in +//! its own step still goes out with that step. use std::time::Duration; @@ -157,19 +158,41 @@ fn an_effect_orphaned_in_its_own_step_still_goes_out_with_that_step() { } #[test] -fn teardown_orphans_every_awaited_effect_and_withdraws_none() { +fn teardown_orphans_every_awaited_effect_and_withdraws_every_outstanding_one() { let mut table = EffectTable::default(); + let earlier = issue_timer(&mut table, chain(2)); let (call, _) = issue_tool_call(&mut table, chain(0)); let timer = issue_timer(&mut table, chain(1)); + let answered = issue_timer(&mut table, chain(1)); + assert_eq!(table.orphan(earlier), None); + assert!(matches!(table.arrive(answered), Arrival::Awaited(_))); + table.orphan(call); table.orphan_all(); let (_, withdrawn) = table.take_step(); - assert!(withdrawn.is_empty(), "{withdrawn:?}"); + assert_eq!( + withdrawn, + [earlier, call, timer], + "every effect still owed an answer, an earlier orphan included, once each, in id order" + ); assert!(!table.has_awaited()); + assert!(matches!(table.arrive(earlier), Arrival::Orphaned)); assert!(matches!(table.arrive(call), Arrival::Orphaned)); assert!(matches!(table.arrive(timer), Arrival::Orphaned)); assert!(table.all_answered()); } +#[test] +fn a_step_lists_no_withdrawal_already_answered() { + let mut table = EffectTable::default(); + let (answered, _) = issue_tool_call(&mut table, chain(0)); + let (owed, _) = issue_tool_call(&mut table, chain(0)); + table.orphan(answered); + table.orphan(owed); + assert!(matches!(table.arrive(answered), Arrival::Orphaned)); + let (_, withdrawn) = table.take_step(); + assert_eq!(withdrawn, [owed], "the caller already answered {answered}"); +} + #[test] fn awaited_by_lists_one_chains_awaited_effects_in_issue_order() { let mut table = EffectTable::default(); diff --git a/crates/runvil-internal/engine/src/execute/scheduler/effects.rs b/crates/runvil-internal/engine/src/execute/scheduler/effects.rs index 28c6c369a..391abff38 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/effects.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/effects.rs @@ -7,8 +7,10 @@ //! to the scheduler, which fails the run loudly rather than passing it //! silently. An effect issued in a step goes out with that step even when //! the run's end orphaned it in the same step, because the caller still -//! owes it an answer. The step also lists the tool calls orphaned since -//! the last step, in id order. +//! owes it an answer. The step also lists the effects withdrawn since the +//! last step, in id order: a tool call orphaned while the run goes on, and +//! every outstanding effect, of every kind, once the run's end tears it +//! down. //! //! An awaited entry holds what its chain asked for, in the terms //! `apply_answer` needs to turn the caller's raw [`EffectAnswer`] into the @@ -178,7 +180,7 @@ pub(super) struct EffectTable { /// The effects issued since the step began, in issue order, each with /// the provenance of the task that built it; the step returns them. issued: Vec, - /// Tool-call effects orphaned since the last step, drained into + /// Effects withdrawn since the last step, drained into /// `Step::Pending`. withdrawn: Vec, /// The next effect id: a run-wide counter, so every effect the run @@ -250,12 +252,15 @@ impl EffectTable { Some((chain, call.exec)) } - /// Orphans every awaited effect at the run's teardown, silently: no - /// call is listed as withdrawn and no identity is returned, because + /// Orphans every awaited effect at the run's teardown and withdraws + /// every outstanding effect, the earlier orphans included, so the + /// caller may drop all it holds. No identity is returned, because /// those end when the run's scope closes. pub(super) fn orphan_all(&mut self) { - for entry in self.outstanding.values_mut() { + self.withdrawn.clear(); + for (id, entry) in &mut self.outstanding { *entry = Outstanding::Orphaned; + self.withdrawn.push(*id); } } @@ -296,23 +301,16 @@ impl EffectTable { } /// Takes the step's outgoing queue: the effects issued since the step - /// began, in issue order, and the tool calls withdrawn since the last - /// step, in id order. + /// began, in issue order, and the effects withdrawn since the last + /// step that still owe their answer, in id order. pub(super) fn take_step(&mut self) -> (Vec, Vec) { let issued = std::mem::take(&mut self.issued); let mut withdrawn = std::mem::take(&mut self.withdrawn); + withdrawn.retain(|id| self.outstanding.contains_key(id)); withdrawn.sort_unstable(); (issued, withdrawn) } - /// Drops the withdrawals listed since the last step. `drive::end` calls - /// this after its teardown: the teardown orphans whatever is still in - /// flight silently, and those orphans are not listed, so a decided run - /// lists none rather than some. - pub(super) fn clear_withdrawn(&mut self) { - self.withdrawn.clear(); - } - /// Whether `id` is recorded as orphaned. #[cfg(test)] pub(super) fn is_orphaned(&self, id: EffectId) -> bool { diff --git a/crates/runvil-internal/engine/src/execute/scheduler/h1.rs b/crates/runvil-internal/engine/src/execute/scheduler/h1.rs index 004fa9dca..1c1df706f 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/h1.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/h1.rs @@ -51,6 +51,7 @@ impl Scheduler { ChainKind::H1, Counters::default(), self.ctx.clone(), + self.ctx.default_binding(), Arc::new(access), SectionPos::top(SectionIndex::FIRST), serde_json::json!({}), @@ -96,6 +97,8 @@ impl Scheduler { // lowered it with `tasks.concurrency`. let counters = chain.counters; let concurrency = chain.concurrency; + // The walk starts on the model the pass ended on. + let model = chain.current_model()?; if crate::parser::detail::sections(self.ctx.prompt()).is_empty() { // No walk follows, so the pass's end is the run's end: a task // the pass spawned and left live ends here under the same @@ -112,6 +115,7 @@ impl Scheduler { ChainKind::Root, counters, walk_ctx, + model, walk_access, start, var, diff --git a/crates/runvil-internal/engine/src/execute/scheduler/tasks-spawn.rs b/crates/runvil-internal/engine/src/execute/scheduler/tasks-spawn.rs index a87df5123..b9871246c 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/tasks-spawn.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/tasks-spawn.rs @@ -95,6 +95,10 @@ impl Scheduler { // A spawned chain's effective limit starts with its spawner's: // its tasks run within the spawner's share and every ancestor's. let child_concurrency = chain.concurrency; + // The task starts on a copy of its spawner's current model, taken + // now, at dispatch: a queued task holds no VM until it is admitted, + // so a copy taken then would see a later switch. + let model = chain.current_model()?; let spawner_access = Arc::clone(chain.access()?); // `chain`'s arena borrow ends here; the resolution names the // target's slice by path, resolved against the shared tree. @@ -135,6 +139,7 @@ impl Scheduler { lineage, counters: Counters::default(), ctx: child_ctx, + model, access: Arc::new(access), start: resolved.pos, kind: ChainKind::Task, diff --git a/crates/runvil-internal/engine/src/execute/scheduler/test_hooks.rs b/crates/runvil-internal/engine/src/execute/scheduler/test_hooks.rs index b80c70e97..4258b197c 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/test_hooks.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/test_hooks.rs @@ -56,6 +56,7 @@ impl Scheduler { lineage, counters: Counters::default(), ctx: self.ctx.clone(), + model: self.ctx.default_binding(), access: Arc::new(access), start: SectionPos::top(SectionIndex::FIRST), kind: ChainKind::Task, @@ -104,6 +105,11 @@ impl Scheduler { self.ctx.turns().load(Ordering::Relaxed) } + /// Takes the error behind the failed outcome the last `Done` carried. + pub(crate) fn take_failure_for_test(&mut self) -> Option { + self.failure.take() + } + /// Starts a chain over the top-level slice at `index`, as a jump out /// of the H1 pass starts the walk, and returns the section name the /// chain reports before its first entry. @@ -165,6 +171,7 @@ impl Scheduler { lineage: ChainId::root(), counters: Counters::default(), ctx: self.ctx.clone(), + model: self.ctx.default_binding(), access: Arc::new(access), start: SectionPos::top(SectionIndex::new(index)), kind: ChainKind::Root, diff --git a/crates/runvil-internal/engine/src/execute/scheduler/tool_call.rs b/crates/runvil-internal/engine/src/execute/scheduler/tool_call.rs index ba9d5aa67..724dd4244 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/tool_call.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/tool_call.rs @@ -157,7 +157,8 @@ impl ChainOps<'_> { /// /// # Errors /// Returns [`Error::UnboundToolCall`] for a name that resolves to no - /// tool. + /// tool, and [`Error::ModelOnlyToolCall`] for a script's call of a + /// catalog tool marked model-only. fn tool_route(&self, name: &str, site: &CallSite) -> Result { let chain = &*self.chain; let vm = chain.frame()?.vm(); @@ -177,11 +178,25 @@ impl ChainOps<'_> { { return Ok(ToolRoute::Adhoc(binding)); } - ToolId::parse(name) + let Some(info) = ToolId::parse(name) .ok() .and_then(|tool| tool_set.catalog().get(&tool)) - .map(|info| ToolRoute::Catalog(ToolBinding::from_info(&info.id.to_string(), info))) - .ok_or_else(|| unbound_tool_call(tool_set, name)) + else { + return Err(unbound_tool_call(tool_set, name)); + }; + // A script cannot know the arguments a render shapes for a model, + // so a model-only tool refuses the call while the catalog keeps + // naming it. + if info.model_only_call { + return Err(crate::Error::ModelOnlyToolCall { + name: name.to_owned(), + id: info.id.to_string(), + }); + } + Ok(ToolRoute::Catalog(ToolBinding::from_info( + &info.id.to_string(), + info, + ))) } /// The turn every report of a call at `site` carries: the requesting @@ -222,8 +237,7 @@ impl ChainOps<'_> { .map(|tool| tool.id().clone()) .collect() }; - let ctx = chain.ctx.clone(); - chain.frame_mut()?.tool_call_counts(&ctx, &ids) + chain.frame_mut()?.tool_call_counts(&ids) } /// Fills `slot` of the batch the chain is parked on with a landed diff --git a/crates/runvil-internal/engine/src/execute/scheduler/walk.rs b/crates/runvil-internal/engine/src/execute/scheduler/walk.rs index 919005d99..43cefe92a 100644 --- a/crates/runvil-internal/engine/src/execute/scheduler/walk.rs +++ b/crates/runvil-internal/engine/src/execute/scheduler/walk.rs @@ -56,6 +56,7 @@ impl Scheduler { ChainKind::Root, after_h1, self.ctx.clone(), + self.ctx.default_binding(), Arc::new(access), SectionPos::top(SectionIndex::FIRST), serde_json::json!({}), @@ -106,8 +107,13 @@ impl ChainOps<'_> { // section - and no SECTION_STARTED, the pass is not a walked // section. Its id is the root chain's entry 0. let section_id = chain.next_entry_id()?; - let frame = - SectionContext::new_h1(&chain.ctx, chain.access()?, §ion_id, &chain.task)?; + let frame = SectionContext::new_h1( + &chain.ctx, + chain.access()?, + &chain.selection, + §ion_id, + &chain.task, + )?; chain.entered = Some(Entered::new(frame)); return Ok(true); } @@ -124,6 +130,7 @@ impl ChainOps<'_> { let frame = SectionContext::new( &chain.ctx, chain.access()?, + &chain.selection, chain.walk.current(), §ion_id, &task, diff --git a/crates/runvil-internal/engine/src/execute/scope.rs b/crates/runvil-internal/engine/src/execute/scope.rs index 6e6373a3f..e5a5be646 100644 --- a/crates/runvil-internal/engine/src/execute/scope.rs +++ b/crates/runvil-internal/engine/src/execute/scope.rs @@ -173,10 +173,10 @@ struct Laid { /// offered into. Every rendered tool's id starts with its Pack's id, a /// rendered run catalog tool is one the section offered in that set, and /// a tool repeated within a set keeps its first copy. Each tool binds -/// under its wire name, and a catalog id keeps the catalog's output -/// kind, listing flag, and stop flag, so a render changes only a catalog -/// tool's wire name, description, and schema, while a model-only tool -/// binds with the flags its render gave it. +/// under its wire name, and a catalog id keeps the catalog's flags and +/// output kind, so a render changes only a catalog tool's wire name, +/// description, and schema, while a model-only tool binds with the flags +/// its render gave it. /// - Within a set, a Pack tool whose wire name an adhoc tool of that set /// takes is dropped. /// - A Dynamic Set tool whose id the Static Set holds is dropped. @@ -335,11 +335,12 @@ fn lay_out( } /// Binds a rendered tool under its wire name, with the catalog's output -/// kind, listing flag, and stop flag when its id is a catalog tool's. +/// kind and flags when its id is a catalog tool's. fn bind(mut tool: ToolInfo, catalog: &ToolCatalog) -> ToolBinding { if let Some(known) = catalog.get(&tool.id) { tool.structured_output = known.structured_output; tool.describes_tools = known.describes_tools; + tool.model_only_call = known.model_only_call; tool.survives_stop = known.survives_stop; } ToolBinding::from_info(&wire_name_of(&tool), &tool) diff --git a/crates/runvil-internal/engine/src/execute/section_context-construct.rs b/crates/runvil-internal/engine/src/execute/section_context-construct.rs index 1142d728c..cd8f1720a 100644 --- a/crates/runvil-internal/engine/src/execute/section_context-construct.rs +++ b/crates/runvil-internal/engine/src/execute/section_context-construct.rs @@ -7,17 +7,18 @@ //! between the two constructors are the list on //! [`SectionContext::new_h1`]. -use std::sync::Arc; +use std::sync::{Arc, Mutex}; use runvil_types::event::lifecycle; use runvil_types::ids::TaskId; use runvil_vfs::Access; -use crate::execute::context::RunState; +use crate::execute::context::ChainState; use crate::execute::section_vm::{self, VmSeed}; use crate::execute::walk_target::{self, RequireOrigin}; use crate::heading_address::{SectionPos, Visible}; use crate::lua::{Library, LibraryKey, SectionVm, Uninjected}; +use crate::model::ModelBinding; use crate::parser::Prompt; use crate::{Error, Result}; @@ -40,7 +41,7 @@ fn library_resolver( /// /// # Errors /// Returns the [`Error`] of the VM construction or the limits install. -fn limited_vm(ctx: &RunState, name: &str) -> Result> { +fn limited_vm(ctx: &ChainState, name: &str) -> Result> { let vm = SectionVm::new_for_section( ctx.nonce(), ctx.tool_set(), @@ -82,9 +83,14 @@ impl SectionContext { /// Returns [`Error::Internal`] when `pos` is past its slice, and /// otherwise the [`Error`] of whichever step failed (see [`section_vm`] /// for which failures tear the VM down). + #[expect( + clippy::too_many_arguments, + reason = "the frame takes the two chain-owned handles (access and selection) beside the section entry's own deltas, and each stays a separate borrow" + )] pub(crate) fn new( - ctx: &RunState, + ctx: &ChainState, access: &Arc, + selection: &Arc>>, pos: &SectionPos, section_id: &str, task_id: &TaskId, @@ -126,6 +132,7 @@ impl SectionContext { item: seed.item.as_ref(), }, access, + selection, section.name(), false, ); @@ -156,8 +163,9 @@ impl SectionContext { /// Returns the [`Error`] of whichever step failed (see [`section_vm`] /// for which failures tear the VM down). pub(crate) fn new_h1( - ctx: &RunState, + ctx: &ChainState, access: &Arc, + selection: &Arc>>, section_id: &str, task_id: &TaskId, ) -> Result { @@ -177,7 +185,7 @@ impl SectionContext { // describes; the executor reads the value back at the freeze (see // the scheduler's H1-to-walk handoff). let sys_json = sys.to_json(); - let setup = ctx.vm_setup(&sys_json, VmSeed::default(), access, title, true); + let setup = ctx.vm_setup(&sys_json, VmSeed::default(), access, selection, title, true); // Setup runs on the bare VM and tears it down itself on failure: // the frame does not exist yet, so its `Drop` cannot own this path. let vm = section_vm::setup_section_vm(vm, &setup, read, resolve, render)?; @@ -190,7 +198,7 @@ impl SectionContext { vm: SectionVm, sys: SectionSys, item: Option, - ctx: &RunState, + ctx: &ChainState, ) -> Self { Self { vm: Some(vm), diff --git a/crates/runvil-internal/engine/src/execute/section_context.rs b/crates/runvil-internal/engine/src/execute/section_context.rs index 00648d03f..ca86f2de4 100644 --- a/crates/runvil-internal/engine/src/execute/section_context.rs +++ b/crates/runvil-internal/engine/src/execute/section_context.rs @@ -14,7 +14,7 @@ //! boundary. //! //! The run-scoped inputs (bindings, models, limits, the shared tools) -//! arrive through the [`RunState`]. +//! arrive through the [`ChainState`]. #[path = "section_context-construct.rs"] mod construct; @@ -28,7 +28,7 @@ use runvil_types::tools::ToolId; use crate::lua::{ProseState, ScriptReport, SectionVm, ToolCallCounts}; use crate::{Error, Result, subst}; -use super::context::{ReportingHandles, RunState}; +use super::context::{ChainState, ReportingHandles}; /// The seeds a spawned task chain's first section entry receives beyond the /// shared Engine values: `tasks.spawn`'s `opts.item` (installed as the @@ -200,7 +200,7 @@ impl SectionContext { /// /// # Errors /// Returns [`Error::Lua`] if the guard cannot be installed. - pub(super) fn install_lazy_prose(&self, ctx: &RunState, template: &str) -> Result<()> { + pub(super) fn install_lazy_prose(&self, ctx: &ChainState, template: &str) -> Result<()> { let template = template.to_owned(); let render = prose_renderer(ctx, self.item.clone()); self.vm() @@ -230,18 +230,14 @@ impl SectionContext { /// # Errors /// Returns the [`Error`] of the first-call install or the /// id seeding. - pub(super) fn tool_call_counts( - &mut self, - ctx: &RunState, - ids: &[ToolId], - ) -> Result { + pub(super) fn tool_call_counts(&mut self, ids: &[ToolId]) -> Result { let Self { vm, sys, counts, .. } = self; let vm = live_vm(vm.as_ref()); let counts = match counts { Some(counts) => counts, - None => counts.insert(install_first_call_state(vm, ctx, sys, ids)?), + None => counts.insert(install_first_call_state(vm, sys, ids)?), }; for id in ids { counts.ensure(&id.to_string())?; @@ -285,7 +281,7 @@ fn live_vm(vm: Option<&SectionVm>) -> &SectionVm { /// the run's `args` and `argv`, shared with the run rather than copied, /// the frame's `item`, and the section state the first read snapshots. fn prose_renderer( - ctx: &RunState, + ctx: &ChainState, item: Option, ) -> impl Fn(&str, ProseState<'_>) -> mlua::Result + Send + Sync + 'static { let raw_args = Arc::clone(ctx.args()); @@ -305,11 +301,11 @@ fn prose_renderer( } /// Installs what a section's first tool call fixes: the counts behind -/// `tools.stats` and the model binding (`sys.model`). +/// `tools.stats` and the current model (`sys.model`). /// /// The counts install backs the Lua `tools.stats` table, seeded with -/// `ids`, and the model resolution freezes the section's binding and -/// sets `sys.model` to the bound model's name. The caller stores the +/// `ids`, and the model read takes the section's current model and +/// sets `sys.model` to its name. The caller stores the /// returned counts, so its slot is set only once every step has /// succeeded, and a failed install runs again on the section's next tool /// call. `sys` changes only once the VM's re-seal succeeds. @@ -319,12 +315,11 @@ fn prose_renderer( /// the `sys` re-seal. fn install_first_call_state( vm: &SectionVm, - ctx: &RunState, sys: &mut SectionSys, ids: &[ToolId], ) -> Result { let counts = vm.install_tool_call_counts(ids)?; - let resolved_model = crate::lua::resolve_model_binding(ctx.models(), vm.model_selection())?; + let resolved_model = vm.current_model()?; if let Some(binding) = resolved_model.as_ref() { let mut enriched = sys.clone(); enriched.model = Some(binding.id().name().to_owned()); diff --git a/crates/runvil-internal/engine/src/execute/section_vm.rs b/crates/runvil-internal/engine/src/execute/section_vm.rs index 9df812bd7..95e603a4d 100644 --- a/crates/runvil-internal/engine/src/execute/section_vm.rs +++ b/crates/runvil-internal/engine/src/execute/section_vm.rs @@ -17,13 +17,14 @@ //! consumes the uninjected VM, so a setup failure tears it down here, the //! one place a VM that never reached its frame is torn down. -use std::sync::Arc; +use std::sync::{Arc, Mutex}; use runvil_types::emitter::Emitter; use runvil_types::packs::Prelude; use runvil_vfs::Access; use crate::lua::{Library, LibraryKey, ProseState, SectionText, SectionVm, Uninjected}; +use crate::model::ModelBinding; use crate::{Error, Result}; /// What a section VM is seeded with beyond the shared Engine values. @@ -49,7 +50,7 @@ pub(crate) struct VmSeed<'a> { pub(crate) struct SectionVmSetup<'a> { /// The run's argument string, installed as the `args` global. pub(crate) args: &'a str, - /// The run's `argv` (`RunState::argv`; `None` installs nil), + /// The run's `argv` (`ChainState::argv`; `None` installs nil), /// with its mode: writable for the H1 pass alone, so the repair pattern /// can assign it, and frozen for every other section. The caller picks /// the mode at construction, so a writable value cannot reach a @@ -63,6 +64,10 @@ pub(crate) struct SectionVmSetup<'a> { /// (a fanout arm included). The store closures share it, so every /// store op is attributed to the chain step's identity. pub(crate) access: &'a Arc, + /// The section's current-model selection: it starts on the `default` + /// binding, or on nothing when the caller supplied no model, and + /// `models.use` rewrites it. + pub(crate) selection: &'a Arc>>, /// The frame constructor's seed: the chain's `var`, plus the item /// seeded into this section on a spawned task's first entry. pub(crate) seed: VmSeed<'a>, @@ -71,8 +76,8 @@ pub(crate) struct SectionVmSetup<'a> { pub(crate) emitter: &'a Emitter, /// The section name used in observations and error messages. pub(crate) section_name: &'a str, - /// The run's tool-loop cap, as `RunState::max_tool_iterations` resolves - /// it, captured by the `models.loop` shim as its round cap. + /// The run's tool-loop cap, as `ChainState::max_tool_iterations` + /// resolves it, captured by the `models.loop` shim as its round cap. pub(crate) max_tool_iterations: usize, /// The run's application-state snapshot, when the caller supplied one: /// its presence gives the section VM the `ui()` global and the @@ -139,6 +144,7 @@ where setup.access, setup.seed.var, setup.argv, + setup.selection, ) .map_err(|failure| Error::from(failure.teardown(setup.emitter, setup.section_name)))?; match install_after_injection(&mut vm, setup, read, resolve, render) { diff --git a/crates/runvil-internal/engine/src/execute/tests.rs b/crates/runvil-internal/engine/src/execute/tests.rs index a320b5371..0bad22a78 100644 --- a/crates/runvil-internal/engine/src/execute/tests.rs +++ b/crates/runvil-internal/engine/src/execute/tests.rs @@ -11,7 +11,6 @@ use runvil_types::untrusted::GuardNonce; use runvil_vfs::{Access, VfsError}; use serde_json::{Value, json}; -use self::context::setup_prompt; use self::fixtures::untrusted_echo_tool; use super::turn::advance_turn; use super::*; @@ -55,19 +54,19 @@ fn run_with_a_pre_cancelled_handle_fails_as_cancelled() { use runvil_types::cancel::CancelHandle; // The explicit-cancel wiring of the public entry point: a handle passed - // through `RunContext::with_cancel` is installed around the whole run body, so + // through `RunConfig::with_cancel` is installed around the whole run body, so // the section's Lua instruction hook observes it and the run ends // cancelled. let source = prompt("", &[("Loop", "local n = 0\nwhile true do n = n + 1 end")]); let handle = CancelHandle::new(); handle.cancel(); let trace = Scenario::new(source) - .context(move |ctx| ctx.with_cancel(handle)) + .config(move |ctx| ctx.with_cancel(handle)) .run(); assert!( - matches!(trace.result(), RunResult::Cancelled), + matches!(trace.outcome(), RunOutcome::Cancelled), "a pre-cancelled handle must end the run cancelled, got {:?}", - trace.result() + trace.outcome() ); } @@ -119,10 +118,13 @@ fn untrusted_nonce_differs_across_runs_under_different_seeds() { // different nonces, so an envelope's tag stays unguessable from one run // to the next as long as the caller draws each seed afresh. (Under one // seed the two runs agree byte for byte, which `run_inputs` pins.) - let source = setup_prompt(PromptSource::new().front("uses: [tools]").section( - "Only", - "return tools.call('tools/untrusted_echo', { value = 'hi' })", - )); + let source = PromptSource::new() + .front("uses: [tools]") + .section( + "Only", + "return tools.call('tools/untrusted_echo', { value = 'hi' })", + ) + .build(); let mut rng = Seeded::new(0x5EED); let mut run_nonces = Vec::new(); for seed in [rng.run_seed(), rng.run_seed()] { diff --git a/crates/runvil-internal/engine/src/execute/tests/adhoc_tools.rs b/crates/runvil-internal/engine/src/execute/tests/adhoc_tools.rs index c2b60d8e4..fd32d95b0 100644 --- a/crates/runvil-internal/engine/src/execute/tests/adhoc_tools.rs +++ b/crates/runvil-internal/engine/src/execute/tests/adhoc_tools.rs @@ -145,9 +145,9 @@ fn cancel_during_a_looping_adhoc_tool_handler_returns_promptly() { .cancel_when(|effect| matches!(effect, Effect::Chat { .. })) .run(); assert!( - matches!(trace.result(), RunResult::Cancelled), + matches!(trace.outcome(), RunOutcome::Cancelled), "expected the cancelled run, got {:?}", - trace.result() + trace.outcome() ); } diff --git a/crates/runvil-internal/engine/src/execute/tests/chat_arm.rs b/crates/runvil-internal/engine/src/execute/tests/chat_arm.rs index c380d5f7a..34a9b5941 100644 --- a/crates/runvil-internal/engine/src/execute/tests/chat_arm.rs +++ b/crates/runvil-internal/engine/src/execute/tests/chat_arm.rs @@ -221,7 +221,7 @@ fn an_out_of_scope_tool_name_fails_the_round_with_out_of_scope_tool() { .offer("echo", echo_tool()) .chat(&gateway) .run(); - match trace.error().inner() { + match trace.error() { Error::OutOfScopeToolCall { name, global_exists, diff --git a/crates/runvil-internal/engine/src/execute/tests/chat_record_rebuild.rs b/crates/runvil-internal/engine/src/execute/tests/chat_record_rebuild.rs index a233907c9..edf64e769 100644 --- a/crates/runvil-internal/engine/src/execute/tests/chat_record_rebuild.rs +++ b/crates/runvil-internal/engine/src/execute/tests/chat_record_rebuild.rs @@ -59,7 +59,7 @@ fn echo_rounds(lua: &str, replies: Vec) -> (Scenario, SentLog) { other => panic!("only model rounds reach the model: {other:?}"), }; rounds.borrow_mut().push(Sent { - record: effect.record(), + record: effect.to_record(), live: messages .iter() .map(|message| serde_json::to_value(message).expect("a message serializes")) diff --git a/crates/runvil-internal/engine/src/execute/tests/compaction.rs b/crates/runvil-internal/engine/src/execute/tests/compaction.rs index 7fb0866fe..87e3a0ae9 100644 --- a/crates/runvil-internal/engine/src/execute/tests/compaction.rs +++ b/crates/runvil-internal/engine/src/execute/tests/compaction.rs @@ -9,7 +9,7 @@ use runvil_parser::test_support::{one_section, prompt}; use super::fixtures::echo_tool; use super::gateway::overflowing; -use super::models_loop::{catalog_scenario, loop_scenario}; +use super::models_loop::loop_scenario; use super::*; use crate::model::CompletionErrorKind; @@ -199,10 +199,7 @@ fn a_compactor_that_calls_a_tool_gets_its_result() { end) return got .. '|' .. messages[1].content .. '|' .. messages[2].content", ); - let trace = catalog_scenario(source) - .tool(echo_tool()) - .chat(&gateway) - .run(); + let trace = loop_scenario(source).tool(echo_tool()).chat(&gateway).run(); assert_eq!(trace.text(), "echoed: precheck|echoed: precheck|done"); } diff --git a/crates/runvil-internal/engine/src/execute/tests/context.rs b/crates/runvil-internal/engine/src/execute/tests/context.rs index 98e223ddb..120192720 100644 --- a/crates/runvil-internal/engine/src/execute/tests/context.rs +++ b/crates/runvil-internal/engine/src/execute/tests/context.rs @@ -1,4 +1,4 @@ -//! Shared context builders, prompt sources, parsers, stores, task ids, and +//! Shared config builders, prompt sources, parsers, stores, task ids, and //! recorders for the execution suites. use runvil_parser::test_support::PromptSource; @@ -7,26 +7,6 @@ use runvil_types::ids::TaskId; use super::*; use crate::test_support::scenario::{EXECUTION, TEST_SEED, TEST_STARTED_AT}; -/// Declares the `writer` role in `source`'s frontmatter and parks it as the -/// prompt-wide default in its H1, so a bound scenario runs its sections -/// under the test model. -pub(super) fn with_writer(source: PromptSource) -> PromptSource { - source - .front("models: { writer: {} }") - .h1("models.default('writer')") -} - -/// The prompt `source` builds, declaring the `writer` role and requiring -/// `## setup` from its H1: the library, after `source`'s sections, that -/// parks the role as the prompt-wide default. -pub(super) fn setup_prompt(source: PromptSource) -> String { - source - .front("models: { writer: {} }") - .h1("require '## setup'") - .section("setup", "models.default('writer')") - .build() -} - /// The scaffold the fanout-arm capability tests share: a `## Parent` that /// fans `### Worker` out over one `alpha` member and returns the first arm's /// text. The worker body and its sibling sections follow. @@ -55,36 +35,22 @@ pub(super) fn sequence(trace: &Trace) -> Vec<(String, String)> { .collect() } -/// The context edit binding the role `label` to `model` by hand, for a run -/// that skips the prepare. -pub(super) fn bind_role( - label: &str, - model: ModelInfo, -) -> impl FnOnce(RunContext) -> RunContext + 'static { - let label = label.to_owned(); - move |mut ctx| { - ctx.model_bindings.bind(&label, model); - ctx - } -} - -/// A [`RunContext`] for the run `name` under the fixed test inputs. -pub(super) fn test_context(name: impl Into) -> RunContext { - RunContext::new(name, TEST_SEED, TEST_STARTED_AT) +/// A [`RunConfig`] for the run `name` under the fixed test inputs. +pub(super) fn test_config(name: impl Into) -> RunConfig { + RunConfig::new(name, TEST_SEED, TEST_STARTED_AT) } /// Compile-time proof that the public execution types are thread-safe. /// -/// `RunContext` holds `Arc` / `Arc` (shared +/// `RunConfig` holds `Arc` / `Arc` (shared /// trait objects) and must be `Send + Sync + 'static` to cross the run's task /// boundaries; the typed error/limit/result surfaces must be too. const fn _public_execution_types_are_send_sync_static() { const fn assert_send_sync_static() {} const fn assert_send_sync() {} - assert_send_sync_static::(); + assert_send_sync_static::(); assert_send_sync_static::(); - assert_send_sync_static::(); - assert_send_sync_static::(); + assert_send_sync_static::(); assert_send_sync_static::(); } diff --git a/crates/runvil-internal/engine/src/execute/tests/debug_and_counts-infer-rounds.rs b/crates/runvil-internal/engine/src/execute/tests/debug_and_counts-infer-rounds.rs index af034bdd8..f65876666 100644 --- a/crates/runvil-internal/engine/src/execute/tests/debug_and_counts-infer-rounds.rs +++ b/crates/runvil-internal/engine/src/execute/tests/debug_and_counts-infer-rounds.rs @@ -5,7 +5,6 @@ use runvil_parser::test_support::PromptSource; use super::*; -use crate::execute::tests::context::setup_prompt; use crate::test_support::recording::{ReplyReport, ThinkingReport}; /// Records an infer round's boundary observations and content hooks as one @@ -82,9 +81,12 @@ fn line_index(lines: &[String], needle: &str) -> usize { #[track_caller] fn run_infer_round(reply: ScriptedReply) -> (String, Vec) { let gateway = ScriptedChat::new(vec![reply]); - let source = setup_prompt( - PromptSource::new().section("Only", "return models.get('writer'):infer('say hello')"), - ); + let source = PromptSource::new() + .section( + "Only", + "return models.infer(models.get('default'), 'say hello')", + ) + .build(); let trace = Scenario::new(source).bound().chat(&gateway).run(); let recorder = InferRoundRecorder::default(); trace.replay(&recorder); diff --git a/crates/runvil-internal/engine/src/execute/tests/debug_and_counts.rs b/crates/runvil-internal/engine/src/execute/tests/debug_and_counts.rs index 0e10be59e..72df3675d 100644 --- a/crates/runvil-internal/engine/src/execute/tests/debug_and_counts.rs +++ b/crates/runvil-internal/engine/src/execute/tests/debug_and_counts.rs @@ -2,34 +2,34 @@ //! infer-round reporting cases sit in `infer_rounds`. use runvil_parser::test_support::{PromptSource, one_section}; -use runvil_types::emitter::DebugMode; -use super::context::{arm_fanout, setup_prompt, with_writer}; +use super::context::arm_fanout; use super::fixtures::{echo_tool, failing_tool, scoped_tool}; use super::models_loop::loop_scenario; use super::*; use crate::test_support::recording::DebugEvent; use crate::test_support::scenario::EXECUTION; -/// The prompt whose one section, `Only`, asks the `writer` role about its +/// The prompt whose one section, `Only`, asks the `default` role about its /// prose. fn ask_the_model() -> String { - with_writer(PromptSource::new().prose( - "Only", - "Ask the model. + PromptSource::new() + .prose( + "Only", + "Ask the model. ```lua return models.infer(prose) ```", - )) - .build() + ) + .build() } -/// A bound run of `source` whose context turns debug reporting on. +/// A bound run of `source` whose config turns debug reporting on. fn debugged(source: String) -> Scenario { Scenario::new(source) .bound() - .context(|ctx| ctx.with_report_debug(DebugMode::On)) + .config(|ctx| ctx.with_report_debug(true)) } #[test] @@ -77,11 +77,13 @@ fn nested_model_infer_capture_reaches_the_debug_sink() { // A nested infer called from Lua must route its request/response // capture to the run's owned debug sink instead of dropping it. let gateway = ScriptedChat::new(vec![text("final answer")]); - let source = setup_prompt(PromptSource::new().section( - "Only", - "local text = models.get('writer'):infer('say hello') + let source = PromptSource::new() + .section( + "Only", + "local text = models.infer(models.get('default'), 'say hello') return text", - )); + ) + .build(); let trace = debugged(source).chat(&gateway).run(); assert_eq!(trace.text(), "final answer"); @@ -110,14 +112,16 @@ fn fanout_arm_debug_events_reach_the_run_sink() { // the run's own debug sink, so an arm's model-turn events land on the // run's sink under the worker's section name. let gateway = ScriptedChat::new(vec![text("arm reply")]); - let source = setup_prompt(arm_fanout().prose( - "### Worker", - "Reply about {{ item }}. + let source = arm_fanout() + .prose( + "### Worker", + "Reply about {{ item }}. ```lua return models.infer(prose) ```", - )); + ) + .build(); let trace = debugged(source).chat(&gateway).run(); assert_eq!(trace.text(), "arm reply"); @@ -152,12 +156,12 @@ fn debug_capture_none_changes_nothing() { #[test] fn tool_calls_count_increments_on_successful_dispatch() { let (tool, called) = scoped_tool("echo", "Echo a test value."); - let source = setup_prompt(PromptSource::new().front("uses: [tools]").section( + let source = PromptSource::new().front("uses: [tools]").section( "Only", "tools.call('tools/echo', { value = 'x' }) assert(tools.stats.tools['tools/echo'] == 1, 'expected 1 call, got ' .. tostring(tools.stats.tools['tools/echo'])) return 'ok'", - )); + ).build(); let trace = Scenario::new(source).bound().tool(tool).run(); assert_eq!(trace.text(), "ok"); assert_eq!(called.load(Ordering::SeqCst), 1); @@ -192,7 +196,7 @@ fn tool_calls_count_increments_even_when_tool_errors() { #[test] fn before_the_freeze_a_dynamically_offered_tool_counts_from_zero_and_packs_holds_no_dynamic() { - let source = setup_prompt(PromptSource::new().front("uses: [tools]").section( + let source = PromptSource::new().front("uses: [tools]").section( "Only", "assert(tools.stats == nil, 'tools.stats appears at the first tool call') tools.offer('tools/echo') @@ -200,7 +204,7 @@ fn before_the_freeze_a_dynamically_offered_tool_counts_from_zero_and_packs_holds tools.call('tools/echo', { value = 'x' }) local ok = pcall(function() return tools.stats.packs.dynamic end) return tools.stats.total .. '|' .. tools.stats.packs.tools .. '|' .. tools.stats.tools['tools/lookup'] .. '|' .. tostring(ok)", - )); + ).build(); let (echo, _) = scoped_tool("echo", "Echo a test value."); let (lookup, _) = scoped_tool("lookup", "Look up a value."); let trace = Scenario::new(source).bound().tool(echo).tool(lookup).run(); @@ -270,9 +274,11 @@ fn tool_calls_count_zero_for_uncalled_alias_fails_epilog_assert() { // The first script dispatch installs the counts seeded from the // frozen sets, so an offered but uncalled tool reads as 0 and an // author assert on it fails the run with its own message. - let source = setup_prompt(PromptSource::new().front("uses: [tools]").prose( - "Only", - "```lua + let source = PromptSource::new() + .front("uses: [tools]") + .prose( + "Only", + "```lua tools.offer('tools/search') local _ = tools.call('tools/other', { value = 'x' }) ``` @@ -281,7 +287,8 @@ fn tool_calls_count_zero_for_uncalled_alias_fails_epilog_assert() { assert(tools.stats.tools['tools/search'] > 0, 'search was never called') return 'unreached' ```", - )); + ) + .build(); let (search, _) = scoped_tool("search", "Search for things."); let (other, _) = scoped_tool("other", "Other things."); let trace = Scenario::new(source).bound().tool(search).tool(other).run(); @@ -294,9 +301,11 @@ fn tool_calls_count_zero_for_uncalled_alias_fails_epilog_assert() { #[test] fn tool_calls_typo_alias_is_a_hard_error_with_seeded_set() { - let source = setup_prompt(PromptSource::new().front("uses: [tools]").prose( - "Only", - "```lua + let source = PromptSource::new() + .front("uses: [tools]") + .prose( + "Only", + "```lua tools.offer('tools/search') local _ = tools.call('tools/search', { value = 'x' }) ``` @@ -305,7 +314,8 @@ fn tool_calls_typo_alias_is_a_hard_error_with_seeded_set() { local _ = tools.stats.tools['tools/serach'] return 'unreached' ```", - )); + ) + .build(); let (tool, _) = scoped_tool("search", "Search for things."); let trace = Scenario::new(source).bound().tool(tool).run(); let msg = trace.error().to_string(); @@ -322,19 +332,21 @@ fn tool_calls_typo_alias_is_a_hard_error_with_seeded_set() { #[test] fn handle_infer_returns_text_without_touching_reply_or_sys() { - // The one infer shape: `handle:infer(...)` returns the round's text and never + // The one infer shape: `models.infer(handle, prompt)` returns the round's text and never // sets `reply` or `sys.reply_finish_reason`. let gateway = ScriptedChat::new(vec![text("pong")]); - let source = setup_prompt(PromptSource::new().section( - "Only", - "local text = models.get('writer'):infer('say hello') + let source = PromptSource::new() + .section( + "Only", + "local text = models.infer(models.get('default'), 'say hello') assert(type(text) == 'string', 'infer must return text') assert(text == 'pong') assert(reply == nil, 'infer must not set reply') assert(not pcall(function() return sys.reply_finish_reason end), 'infer must not touch sys') return text", - )); + ) + .build(); let trace = Scenario::new(source).bound().chat(&gateway).run(); assert_eq!(trace.text(), "pong"); let body = gateway diff --git a/crates/runvil-internal/engine/src/execute/tests/discovery.rs b/crates/runvil-internal/engine/src/execute/tests/discovery.rs index c3ca52267..e69d9c9e4 100644 --- a/crates/runvil-internal/engine/src/execute/tests/discovery.rs +++ b/crates/runvil-internal/engine/src/execute/tests/discovery.rs @@ -9,7 +9,7 @@ use runvil_parser::test_support::one_section; use runvil_types::event::Event; use super::fixtures::{echo_tool, scoped_tool}; -use super::models_loop::{LOOP_TO_TEXT, catalog_scenario, loop_model_name}; +use super::models_loop::{LOOP_TO_TEXT, loop_model_name, loop_scenario}; use super::*; use crate::execute::run::{EffectRecord, ToolCallOrigin, ToolCaller}; use crate::test_support::rig::{Rig, RigTrace}; @@ -62,7 +62,7 @@ fn looped( let mut replies: Vec<_> = requested.iter().map(|batch| calls(batch)).collect(); replies.push(text("done")); let gateway = ScriptedChat::new(replies); - let mut scenario = catalog_scenario(one_section(section)) + let mut scenario = loop_scenario(one_section(section)) .tool(echo_tool()) .tool(scoped_tool("lookup", "Look up a value.").0); for tool in extra { @@ -132,7 +132,7 @@ fn issued_calls(rig: &RigTrace) -> Vec { fn model_origin() -> ToolCallOrigin { ToolCallOrigin::new(EXECUTION, "Only", ToolCaller::Model) .with_entry("0.1") - .with_model(loop_model_name("writer")) + .with_model(loop_model_name("default")) } /// A model call of the catalog tool `id` under `alias` with `value`. @@ -462,7 +462,7 @@ fn a_scripts_call_of_a_discovery_tools_id_fails_as_an_unknown_id() { Vec::new(), &[], ); - match rig.trace().error().inner() { + match rig.trace().error() { Error::UnboundToolCall { name, .. } => assert_eq!(name, "dynamic/call"), other => panic!("expected the unbound-tool error, got {other:?}"), } diff --git a/crates/runvil-internal/engine/src/execute/tests/effects.rs b/crates/runvil-internal/engine/src/execute/tests/effects.rs index 5a6c3e542..d89e6105f 100644 --- a/crates/runvil-internal/engine/src/execute/tests/effects.rs +++ b/crates/runvil-internal/engine/src/execute/tests/effects.rs @@ -45,7 +45,7 @@ fn models_infer_issues_exactly_one_chat_effect_over_one_user_message() { rig.issued(), vec![EffectRecord::Chat { round: RoundId::new(0), - alias: "writer".to_owned(), + alias: "default".to_owned(), after: None, keep: 0, messages: vec![json!({ "role": "user", "content": "ask" })], @@ -87,7 +87,7 @@ fn a_models_loop_round_issues_one_chat_effect_and_one_tool_call_effect_per_call( assert_eq!( records[0], EffectRecord::Render { - alias: "writer".to_owned(), + alias: "default".to_owned(), entry: "0.1".to_owned(), offered: vec![PackToolSets::new( PackId::parse("tools").expect("a valid Pack id"), @@ -109,7 +109,7 @@ fn a_models_loop_round_issues_one_chat_effect_and_one_tool_call_effect_per_call( tool: ToolId::parse("tools/echo").expect("a valid id"), alias: "echo".to_owned(), args: json!({ "value": "hi" }), - origin: origin("Only", "0.1", ToolCaller::Model).with_model(loop_model_name("writer")), + origin: origin("Only", "0.1", ToolCaller::Model).with_model(loop_model_name("default")), }, "the model's call is one tool_call effect naming the bound identity and the model" ); diff --git a/crates/runvil-internal/engine/src/execute/tests/event_vendors.rs b/crates/runvil-internal/engine/src/execute/tests/event_vendors.rs index 4e6711f11..6c8b21da0 100644 --- a/crates/runvil-internal/engine/src/execute/tests/event_vendors.rs +++ b/crates/runvil-internal/engine/src/execute/tests/event_vendors.rs @@ -8,8 +8,9 @@ use runvil_types::event::Event; use runvil_types::packs::PackId; use super::fixtures::echo_tool; -use super::models_loop::catalog_scenario; +use super::models_loop::loop_scenario; use super::*; +use crate::lua::RosterEntry; /// A section that offers `tools/echo` and an adhoc `grab`, lets the model /// call both in one batch, then calls `tools/echo` from the script. @@ -22,8 +23,8 @@ const BOTH_CALLERS: &str = "tools.offer('tools/echo') return tools.call('tools/echo', { value = 'script' })"; /// The run of [`BOTH_CALLERS`] with `tools/echo` in the catalog, its -/// context edited by `roster`. -fn both_callers(roster: impl FnOnce(RunContext) -> RunContext + 'static) -> Trace { +/// config edited by `roster`. +fn both_callers(roster: impl FnOnce(RunConfig) -> RunConfig + 'static) -> Trace { let gateway = ScriptedChat::new(vec![ calls(&[ ("call_1", "grab", r#"{"value":"x"}"#), @@ -31,9 +32,9 @@ fn both_callers(roster: impl FnOnce(RunContext) -> RunContext + 'static) -> Trac ]), text("done"), ]); - let trace = catalog_scenario(one_section(BOTH_CALLERS)) + let trace = loop_scenario(one_section(BOTH_CALLERS)) .tool(echo_tool()) - .context(roster) + .config(roster) .chat(&gateway) .run(); assert_eq!(trace.text(), "echoed: script"); @@ -85,8 +86,10 @@ fn result(id: &str, alias: &str, vendor: Option<&str>) -> (String, String, Optio #[test] fn a_catalog_call_records_the_vendor_the_caller_reported_and_an_adhoc_call_none() { let tools = PackId::parse("tools").expect("a static valid Pack id"); - let trace = - both_callers(move |ctx| ctx.with_packs(vec![UsablePack::new(tools, "acme")], vec![])); + let usable = RosterEntry::Usable { + vendor: "acme".to_owned(), + }; + let trace = both_callers(move |ctx| ctx.with_packs([(tools, usable)])); let events = trace.events(); assert_eq!( batch_vendors(events), diff --git a/crates/runvil-internal/engine/src/execute/tests/exec_flow.rs b/crates/runvil-internal/engine/src/execute/tests/exec_flow.rs index 6aca3f5dd..c8bb5b97f 100644 --- a/crates/runvil-internal/engine/src/execute/tests/exec_flow.rs +++ b/crates/runvil-internal/engine/src/execute/tests/exec_flow.rs @@ -2,7 +2,7 @@ use runvil_parser::test_support::{PromptSource, synthetic_section}; -use super::context::{arm_fanout, with_writer}; +use super::context::arm_fanout; use super::*; /// Alternating lua/prose blocks run in order; each Lua block that wants the @@ -11,7 +11,7 @@ use super::*; #[test] fn section_with_alternating_blocks_executes_in_order() { let gateway = ScriptedChat::new(vec![text("reply-1"), text("reply-2")]); - let source = with_writer(PromptSource::new()).prose( + let source = PromptSource::new().prose( "Only", r" ```lua @@ -48,7 +48,7 @@ fn call_runs_named_section_as_subroutine() { // The subroutine sits after its run-ending caller: a contained chain // falls through like any walk, so a subroutine placed before later // sections would pull them into the chain. - let source = with_writer(PromptSource::new()) + let source = PromptSource::new() .section( "Main", " @@ -82,7 +82,7 @@ fn call_runs_named_section_as_subroutine() { #[test] fn nested_call_without_input_inherits_the_chains_args() { let gateway = ScriptedChat::new(vec![text("inner-reply")]); - let source = with_writer(PromptSource::new()) + let source = PromptSource::new() .section( "Main", " @@ -139,23 +139,20 @@ fn duplicate_top_level_section_names_error_loudly() { assert!(rendered.contains("ambiguous"), "error was: {rendered}"); } -/// [`arm_fanout`] under the `writer` role, with a `### Worker` whose Lua is -/// `worker`. -fn writer_arm(worker: &str) -> String { - with_writer(arm_fanout()) - .section("### Worker", worker) - .build() +/// [`arm_fanout`] with a `### Worker` whose Lua is `worker`. +fn default_arm(worker: &str) -> String { + arm_fanout().section("### Worker", worker).build() } -/// The worker Lua that asks the model through the `writer` handle. -const HANDLE_INFER: &str = "return models.get('writer'):infer('ping about ' .. item)"; +/// The worker Lua that asks the model through the `default` handle. +const HANDLE_INFER: &str = "return models.infer(models.get('default'), 'ping about ' .. item)"; -/// `handle:infer(...)` works inside an arm, so a worker's Lua can call the model +/// `models.infer(models.get('default'), ...)` works inside an arm, so a worker's Lua can call the model /// directly. #[test] fn fanout_arm_model_infer_works_inside_an_arm() { let gateway = ScriptedChat::new(vec![text("pong")]); - let trace = Scenario::new(writer_arm(HANDLE_INFER)) + let trace = Scenario::new(default_arm(HANDLE_INFER)) .bound() .chat(&gateway) .run(); @@ -179,14 +176,14 @@ fn fanout_arm_model_infer_works_inside_an_arm() { ); } -/// `handle:infer(...)` inside an arm handed no client surfaces the lazy-creation +/// `models.infer` with a leading handle inside an arm handed no client surfaces the lazy-creation /// error. #[test] fn fanout_arm_model_infer_without_a_client_surfaces_the_disabled_gateway() { // A scenario without a model answers every `Chat` with the disabled // gateway's failure: the round fails with that error and nothing // reaches the network, however the process environment is configured. - let trace = Scenario::new(writer_arm(HANDLE_INFER)).bound().run(); + let trace = Scenario::new(default_arm(HANDLE_INFER)).bound().run(); let rendered = trace.error().to_string(); assert!( rendered.contains("model access is turned off or not configured"), @@ -197,13 +194,17 @@ fn fanout_arm_model_infer_without_a_client_surfaces_the_disabled_gateway() { /// An unknown model alias errors loudly inside an arm, same as on the walk. #[test] fn fanout_arm_model_infer_with_an_unknown_alias_errors_loudly() { - let trace = Scenario::new(writer_arm("return models.get('ghost'):infer('ping')")) - .bound() - .run(); + let trace = Scenario::new(default_arm( + "return models.infer(models.get('ghost'), 'ping')", + )) + .bound() + .run(); let rendered = trace.error().to_string(); assert!( - rendered.contains("models.get alias \"ghost\" is not a bound model role"), - "the unknown alias must be named: {rendered}" + rendered.contains( + "models.get label \"ghost\" is not a model role: expected default, strong, small, fast or creative" + ), + "the unknown label must be named: {rendered}" ); } diff --git a/crates/runvil-internal/engine/src/execute/tests/exit_rules.rs b/crates/runvil-internal/engine/src/execute/tests/exit_rules.rs index 538aa2a0c..a18deee98 100644 --- a/crates/runvil-internal/engine/src/execute/tests/exit_rules.rs +++ b/crates/runvil-internal/engine/src/execute/tests/exit_rules.rs @@ -89,7 +89,7 @@ fn unsupported_major_one_is_refused() { // declared version rather than silently degrading to major 0. let trace = Scenario::new(ran_under("runvil: 1\n")).run(); assert!( - matches!(trace.error().inner(), Error::UnsupportedVersion(1)), + matches!(trace.error(), Error::UnsupportedVersion(1)), "the gate refuses major 1 by name, got {:?}", trace.error() ); @@ -100,7 +100,7 @@ fn unsupported_major_is_refused() { // Any major other than 0 is refused, never silently degraded to major 0. let trace = Scenario::new(ran_under("runvil: 2\n")).run(); assert!( - matches!(trace.error().inner(), Error::UnsupportedVersion(2)), + matches!(trace.error(), Error::UnsupportedVersion(2)), "the gate refuses major 2 by name, got {:?}", trace.error() ); @@ -111,7 +111,7 @@ fn missing_version_is_not_a_runvil_prompt() { // No `runvil:` key: not our prompt, so `run` declines with a Parse // error rather than executing it. let trace = Scenario::new(ran_under("")).run(); - match trace.error().inner() { + match trace.error() { Error::Parse(error) => { assert_eq!(error.kind(), ParseErrorKind::Structure); assert_eq!( @@ -126,6 +126,14 @@ fn missing_version_is_not_a_runvil_prompt() { } other => panic!("expected Parse, got {other:?}"), } + assert_eq!( + trace.outcome(), + &RunOutcome::Failed { + kind: RunErrorKind::Parse, + message: "not a runvil prompt: no runvil version (prompt t)".to_owned(), + }, + "the outcome's message names the prompt the failure's location named" + ); } // --- Cross-section store persistence --- @@ -262,7 +270,7 @@ fn empty_stop_turn_without_tool_calls_fails() { for offered in [None, Some(echo_tool())] { let trace = exit_scenario(vec![text_finish("", "stop")], offered).run(); let events = exit_events(&trace); - match trace.error().inner() { + match trace.error() { Error::Completion(error) => { assert_eq!(error.kind(), CompletionErrorKind::EmptyReply); assert_eq!(error.finish_reason(), Some("stop")); @@ -286,7 +294,7 @@ fn empty_truncated_final_text_fails_without_truncation_detail() { // truncated. let trace = exit_scenario(vec![text_finish("", "length")], None).run(); let events = exit_events(&trace); - match trace.error().inner() { + match trace.error() { Error::Completion(error) => { assert_eq!(error.kind(), CompletionErrorKind::EmptyReply); assert_eq!(error.finish_reason(), Some("length")); @@ -306,7 +314,7 @@ fn empty_turn_without_finish_reason_after_tool_call_fails() { ) .run(); let events = exit_events(&trace); - match trace.error().inner() { + match trace.error() { Error::Completion(error) => { assert_eq!(error.kind(), CompletionErrorKind::EmptyReply); assert_eq!(error.finish_reason(), None); diff --git a/crates/runvil-internal/engine/src/execute/tests/fixtures.rs b/crates/runvil-internal/engine/src/execute/tests/fixtures.rs index a08c29bcf..272446f09 100644 --- a/crates/runvil-internal/engine/src/execute/tests/fixtures.rs +++ b/crates/runvil-internal/engine/src/execute/tests/fixtures.rs @@ -33,6 +33,18 @@ pub(super) fn echo_tool() -> FnTool { .description("Echo the value argument back to the caller.") } +/// The fixture Pack's `tools/replace`, which echoes its `value` argument +/// and which only a model may call. +#[track_caller] +pub(super) fn model_only_tool() -> FnTool { + FnTool::new("tools/replace", value_schema(), |args| { + let value = require_string_arg(&args, "value")?; + Ok(ToolOutput::trusted(format!("replaced: {value}"))) + }) + .description("Replace text the model names.") + .model_only_call() +} + /// The fixture Pack's `tools/untrusted_echo`, which echoes its `value` /// argument as untrusted external data, so its output is guard-wrapped. #[track_caller] diff --git a/crates/runvil-internal/engine/src/execute/tests/full_id_calls.rs b/crates/runvil-internal/engine/src/execute/tests/full_id_calls.rs index 11faf6cd3..7fa9c8bae 100644 --- a/crates/runvil-internal/engine/src/execute/tests/full_id_calls.rs +++ b/crates/runvil-internal/engine/src/execute/tests/full_id_calls.rs @@ -8,7 +8,7 @@ use runvil_parser::test_support::{one_section, prompt}; use runvil_types::event::Event; use super::fixtures::{echo_tool, scoped_tool}; -use super::models_loop::catalog_scenario; +use super::models_loop::loop_scenario; use super::*; use crate::execute::run::{Effect, EffectAnswer, EffectRecord, Step, ToolCallOrigin, ToolCaller}; use crate::test_support::rig::Rig; @@ -30,18 +30,18 @@ fn script_origin() -> ToolCallOrigin { ToolCallOrigin::new(EXECUTION, "Only", ToolCaller::Script).with_entry("0.1") } -/// A run of the one section `lua` prepared against a catalog of the echo -/// fixture and a second tool the prompts never offer. +/// A run of the one section `lua` over a catalog of the echo fixture and a +/// second tool the prompts never offer. #[track_caller] fn echo_and_concrete(lua: &str) -> Scenario { echo_and_concrete_sections(&[("Only", lua)]) } -/// A run of `sections`, each a heading and its Lua, prepared against the -/// catalog [`echo_and_concrete`] uses. +/// A run of `sections`, each a heading and its Lua, over the catalog +/// [`echo_and_concrete`] uses. #[track_caller] fn echo_and_concrete_sections(sections: &[(&str, &str)]) -> Scenario { - catalog_scenario(prompt("", sections)) + loop_scenario(prompt("", sections)) .tool(echo_tool()) .tool(scoped_tool("concrete", "Concrete description.").0) } @@ -61,14 +61,14 @@ fn unofferable_tool() -> FnTool { #[test] fn a_script_calls_effect_record_and_result_name_the_tool_by_its_id() { let source = one_section("return tools.call('tools/echo', { value = 'hi' })"); - let mut run = catalog_scenario(source).tool(echo_tool()).into_run(); + let mut run = loop_scenario(source).tool(echo_tool()).into_run(); let (id, effect) = only_effect(run.step()); let Effect::ToolCall { alias, .. } = &effect else { panic!("the script's call is the one effect: {effect:?}"); }; assert_eq!(alias, ECHO_ID, "the effect names the id as its alias"); assert_eq!( - effect.record(), + effect.to_record(), EffectRecord::ToolCall { tool: echo_id(), alias: ECHO_ID.to_owned(), @@ -80,10 +80,10 @@ fn a_script_calls_effect_record_and_result_name_the_tool_by_its_id() { id, EffectAnswer::ToolCall(Ok(ToolOutput::trusted("echoed: hi"))), ); - let Step::Done { result, events } = run.step() else { + let Step::Done { outcome, events } = run.step() else { panic!("the answered call ends the run"); }; - assert_eq!(text_of(result), "echoed: hi"); + assert_eq!(text_of(outcome), "echoed: hi"); let aliases: Vec<&str> = events .iter() .filter_map(|event| match event { @@ -100,7 +100,7 @@ fn a_script_reaches_by_id_a_catalog_tool_no_round_could_advertise() { "assert(tools.get('hidden/shape') ~= nil, 'the tool is in the run catalog')\n\ return tools.call('hidden/shape', { value = 'hi' })", ); - let rig = Rig::run(catalog_scenario(source).tool(unofferable_tool())); + let rig = Rig::run(loop_scenario(source).tool(unofferable_tool())); assert_eq!(rig.trace().text(), "unoffered: hi"); assert_eq!( rig.issued(), @@ -160,7 +160,7 @@ fn a_script_call_naming_a_string_without_a_slash_raises_before_any_resolution() assert(not wire_ok and not bare_ok, 'neither name is a tool id')\n\ return wire_err.kind .. ':' .. tostring(wire_err) .. '|' .. tostring(bare_err)", ); - let rig = Rig::run(catalog_scenario(source).tool(echo_tool())); + let rig = Rig::run(loop_scenario(source).tool(echo_tool())); assert_eq!( rig.trace().text(), "lua:tools.call: \"tools_echo\" is not a tool id|tools.call: \"echo\" is not a tool id" @@ -195,7 +195,7 @@ fn a_script_call_by_id_and_by_tool_object_issue_the_same_effect() { "return tools.call('tools/echo', { value = 'hi' }) .. '|' .. \ tools.call(tools.get('tools/echo'), { value = 'hi' })", ); - let rig = Rig::run(catalog_scenario(source).tool(echo_tool())); + let rig = Rig::run(loop_scenario(source).tool(echo_tool())); assert_eq!(rig.trace().text(), "echoed: hi|echoed: hi"); let call = EffectRecord::ToolCall { tool: echo_id(), @@ -209,7 +209,7 @@ fn a_script_call_by_id_and_by_tool_object_issue_the_same_effect() { #[test] fn a_name_that_is_no_catalog_id_is_unbound_and_lists_the_catalog_ids() { let rig = Rig::run(echo_and_concrete("return tools.call('tools/missing', {})")); - let error = rig.trace().error().inner(); + let error = rig.trace().error(); match error { Error::UnboundToolCall { name, ids } => { assert_eq!(name, "tools/missing"); @@ -237,7 +237,7 @@ fn a_model_issued_call_never_resolves_an_id() { assert(not ok, 'a model-issued id call is refused')\n\ return err.kind .. ':' .. err.name", ); - let rig = Rig::run(catalog_scenario(source).tool(echo_tool()).raw_shims()); + let rig = Rig::run(loop_scenario(source).tool(echo_tool()).raw_shims()); assert_eq!(rig.trace().text(), "unbound_tool:tools/echo"); assert!(rig.issued().is_empty(), "nothing was issued"); } diff --git a/crates/runvil-internal/engine/src/execute/tests/happens_before.rs b/crates/runvil-internal/engine/src/execute/tests/happens_before.rs index f26cfb579..528631f2a 100644 --- a/crates/runvil-internal/engine/src/execute/tests/happens_before.rs +++ b/crates/runvil-internal/engine/src/execute/tests/happens_before.rs @@ -18,7 +18,7 @@ //! The two concurrency traces sit in `concurrency`. use crate::RunErrorKind; -use crate::execute::RunResult; +use crate::execute::RunOutcome; use crate::test_support::rig::{Rig, RigTrace}; use crate::test_support::scenario::{Batching, TestStore, drive_batched}; use crate::test_support::seeded::Seeded; @@ -265,7 +265,7 @@ fn trace_scenario(source: &str, store: &TestStore) -> Scenario { #[track_caller] fn serial_results( scenario: impl Fn(&TestStore) -> Scenario, -) -> Vec<(Batching, RunResult, TestStore)> { +) -> Vec<(Batching, RunOutcome, TestStore)> { let mut results = Vec::new(); for batching in BATCHINGS { let store = TestStore::new(); @@ -292,18 +292,21 @@ fn shuffled_results(scenario: impl Fn(&TestStore) -> Scenario) -> Vec<(Seeded, R } /// The text of a successful result, or `None`. -fn ok_text(result: &RunResult) -> Option<&str> { +fn ok_text(result: &RunOutcome) -> Option<&str> { match result { - RunResult::Completed(text) => Some(text), + RunOutcome::Completed { final_text: text } => Some(text), _ => None, } } /// Whether a result is the fatal determinism violation. -fn is_determinism(result: &RunResult) -> bool { +fn is_determinism(result: &RunOutcome) -> bool { matches!( result, - RunResult::Failure(error) if error.kind() == RunErrorKind::Determinism + RunOutcome::Failed { + kind: RunErrorKind::Vfs, + message, + } if message.starts_with("store determinism violation: ") ) } @@ -338,7 +341,7 @@ fn assert_every_interleaving_leaves( } } for (seeded, rig) in shuffled_results(&scenario) { - let result = rig.trace().result(); + let result = rig.trace().outcome(); seeded.check_eq(&ok_text(result), &Some(expected), format!("{result:?}")); if let Some((path, contents)) = file { let left = rig.trace().read(path); @@ -362,7 +365,7 @@ fn assert_every_interleaving_conflicts(scenario: impl Fn(&TestStore) -> Scenario ); } for (seeded, rig) in shuffled_results(&scenario) { - let result = rig.trace().result(); + let result = rig.trace().outcome(); seeded.check( is_determinism(result), format!("expected determinism, got {result:?}"), @@ -412,7 +415,7 @@ fn join_any_joins_the_task_it_returns_and_only_that_task() { let gateway = ScriptedChat::new(vec![text("ra"), text("rb")]); let pass = |store: &TestStore| trace_scenario(TRACE_5_PASS, store).chat(&gateway); let legal = - |result: &RunResult| matches!(ok_text(result), Some("written by a" | "written by b")); + |result: &RunOutcome| matches!(ok_text(result), Some("written by a" | "written by b")); for (batching, result, _) in serial_results(pass) { assert!( legal(&result), @@ -420,7 +423,7 @@ fn join_any_joins_the_task_it_returns_and_only_that_task() { ); } for (seeded, rig) in shuffled_results(pass) { - let result = rig.trace().result(); + let result = rig.trace().outcome(); seeded.check(legal(result), format!("an illegal verdict: {result:?}")); } diff --git a/crates/runvil-internal/engine/src/execute/tests/live_infer.rs b/crates/runvil-internal/engine/src/execute/tests/live_infer.rs index 105b312a3..2f5f40561 100644 --- a/crates/runvil-internal/engine/src/execute/tests/live_infer.rs +++ b/crates/runvil-internal/engine/src/execute/tests/live_infer.rs @@ -1,32 +1,25 @@ //! Tests for live `models.infer` against a scripted gateway, including H1 chunks and required libraries. -use runvil_parser::test_support::{PromptSource, parse_ok}; +use runvil_parser::test_support::PromptSource; use runvil_types::cancel::CancelHandle; -use super::context::test_context; use super::fixtures::echo_tool; use super::*; -use crate::test_support::scenario::{EXECUTION, TEST_MODEL, test_model}; -/// The prompt declaring the `writer` role, with no default parked. -fn writer_role() -> PromptSource { - PromptSource::new().front("models: { writer: {} }") -} - -/// The prompt whose H1 infers once through the `writer` handle into +/// The prompt whose H1 infers once through the `default` handle into /// `var.answer`, which its `Result` section returns. fn h1_handle_infer() -> String { - writer_role() + PromptSource::new() .h1(" - local writer = models.default('writer') - var.answer = writer:infer('answer once')") + local writer = models.get('default') + var.answer = models.infer(writer, 'answer once')") .section("Result", "return var.answer") .build() } #[test] -fn the_callers_client_serves_a_run_the_context_never_names() { - // The context is the Engine's input; the scripted model lives on the +fn the_callers_client_serves_a_run_the_config_never_names() { + // The config is the Engine's input; the scripted model lives on the // scenario, which answers the run's completions with it. Nothing about // the gateway crosses the Engine's boundary. let gateway = ScriptedChat::new(vec![text("canned answer")]); @@ -49,7 +42,7 @@ fn unread_h1_prose_stays_inert_and_explicit_infer_requires_a_model() { // whose substitution would fail or stay empty - discards at the pass's // end without requiring a model. Only an explicit `models.infer` of the // prose requires a binding. - let unread = writer_role() + let unread = PromptSource::new() .h1("var.omit = ''") .h1_prose("{{ var.omit }}") .section("Result", "return 'ok'"); @@ -60,14 +53,14 @@ fn unread_h1_prose_stays_inert_and_explicit_infer_requires_a_model() { "unread H1 prose must not require a model" ); - let reading = writer_role() + let reading = PromptSource::new() .h1_prose("ask") .h1("return models.infer(prose)"); let trace = Scenario::new(reading.build()).run(); let error = trace.error(); assert_eq!( error.kind(), - RunErrorKind::Binding, + RunErrorKind::RequirementsUnmet, "expected ModelRequired, got {error}" ); assert!( @@ -78,7 +71,7 @@ fn unread_h1_prose_stays_inert_and_explicit_infer_requires_a_model() { #[test] fn a_library_function_resolves_engine_globals_when_called() { - let source = writer_role() + let source = PromptSource::new() .section("Result", "require '## helpers'\nreturn read_args()") .section("helpers", "function read_args() return args end"); let trace = Scenario::new(source.build()) @@ -94,7 +87,7 @@ fn a_library_calls_engine_globals_at_load_time() { // A library loads into the requiring section's environment with every // Engine global installed, so its top-level code may use `store`, // `log`, and `args` at load. - let source = writer_role() + let source = PromptSource::new() .section( "Result", "require '## loader'\nreturn store.read('loaded.txt')", @@ -106,12 +99,6 @@ fn a_library_calls_engine_globals_at_load_time() { log('library loaded')", ) .build(); - let (_, requirements) = - test_context(EXECUTION).prepare(&parse_ok(&source), Some(test_model(TEST_MODEL))); - assert!( - requirements.is_satisfied(), - "the fixture declares nothing: {requirements:?}" - ); let trace = Scenario::new(source).args("load-time args").bound().run(); assert_eq!(trace.text(), "load-time args"); @@ -128,7 +115,7 @@ fn tool_objects_and_roles_reach_section_call_and_fanout_vms() { // The offering arrives from the catalog: the Pack installs the tool, // and every section VM reads its tool object by id - H1 never runs a // bind. - let source = writer_role() + let source = PromptSource::new() .front("uses: [tools]") .section( "Parent", @@ -147,7 +134,7 @@ fn tool_objects_and_roles_reach_section_call_and_fanout_vms() { .section("Called", "require '## bindings'\nreturn binding_names()") .section( "bindings", - "function binding_names() return tools.get('tools/echo').id .. ':' .. models.get('writer').name end", + "function binding_names() return tools.get('tools/echo').id .. ':' .. models.get('default').name end", ); // The fixture tool's info goes into the catalog and its // implementation answers the run's calls; the run offers every @@ -159,7 +146,7 @@ fn tool_objects_and_roles_reach_section_call_and_fanout_vms() { assert_eq!( trace.text(), - "tools/echo:writer|tools/echo:writer|tools/echo:writer:one,tools/echo:writer:two" + "tools/echo:default|tools/echo:default|tools/echo:default:one,tools/echo:default:two" ); } @@ -169,10 +156,10 @@ fn nested_lua_infer_emits_a_model_turn_observation() { // run's observer, proving owned-observer propagation reaches the nested // inference path. let gateway = ScriptedChat::new(vec![text("pong")]); - let source = writer_role() + let source = PromptSource::new() .h1(" - local writer = models.default('writer') - var.answer = writer:infer('ping')") + local writer = models.get('default') + var.answer = models.infer(writer, 'ping')") .section("Result", "return var.answer"); let trace = Scenario::new(source.build()).bound().chat(&gateway).run(); @@ -201,24 +188,24 @@ fn nested_lua_infer_emits_a_model_turn_observation() { #[test] fn cancelled_nested_infer_does_not_report_model_turn_failed() { let gateway = ScriptedChat::new(vec![text("too late")]); - let source = writer_role().h1(" - local writer = models.default('writer') - return writer:infer('must cancel')"); + let source = PromptSource::new().h1(" + local writer = models.get('default') + return models.infer(writer, 'must cancel')"); let cancel = CancelHandle::new(); let canceller = cancel.clone(); let mut driving = Scenario::new(source.build()) .bound() .chat(&gateway) - .context(move |ctx| ctx.with_cancel(canceller)) + .config(move |ctx| ctx.with_cancel(canceller)) .hold(|effect| matches!(effect, Effect::Chat { .. })) .start(); while driving.step() {} cancel.cancel(); let trace = driving.finish(); assert!( - matches!(trace.result(), RunResult::Cancelled), + matches!(trace.outcome(), RunOutcome::Cancelled), "cancelling an in-flight infer must interrupt the run: {:?}", - trace.result() + trace.outcome() ); assert!( matches!( @@ -240,9 +227,9 @@ fn cancelled_nested_infer_does_not_report_model_turn_failed() { #[test] fn handle_infer_tool_call_violation_uses_entry_point_neutral_wording() { let gateway = ScriptedChat::new(vec![calls(&[("call_1", "ghost", "{}")])]); - let source = writer_role().h1(" - local writer = models.default('writer') - return writer:infer('answer without tools')"); + let source = PromptSource::new().h1(" + local writer = models.get('default') + return models.infer(writer, 'answer without tools')"); let trace = Scenario::new(source.build()).bound().chat(&gateway).run(); let rendered = trace.error().to_string(); assert!( @@ -260,10 +247,8 @@ fn h1_prose_infers_explicitly_and_var_accumulates_into_the_walk() { // The H1 pass reads its pending buffer only through an explicit // infer, and `var` writes accumulate across the pass into the walk. let gateway = ScriptedChat::new(vec![text("final answer")]); - let source = writer_role() - .h1(" - models.default('writer') - var.executions = (var.executions or 0) + 1") + let source = PromptSource::new() + .h1("var.executions = (var.executions or 0) + 1") .h1_prose("Ask for one round.") .h1(" var.first = models.infer(prose) @@ -280,8 +265,7 @@ fn h1_and_h2_prose_each_infer_explicitly_in_source_order() { // The H1 pass and the H2 section each read their own pending // buffer into an explicit infer: two completions, in source order. let gateway = ScriptedChat::new(vec![text("h1 reply"), text("h2 reply")]); - let source = writer_role() - .h1("models.default('writer')") + let source = PromptSource::new() .h1_prose("h1 prose turn") .h1("var.h1 = models.infer(prose)") .prose( diff --git a/crates/runvil-internal/engine/src/execute/tests/model_and_reply-globals.rs b/crates/runvil-internal/engine/src/execute/tests/model_and_reply-globals.rs index 957db67e3..654aadd82 100644 --- a/crates/runvil-internal/engine/src/execute/tests/model_and_reply-globals.rs +++ b/crates/runvil-internal/engine/src/execute/tests/model_and_reply-globals.rs @@ -8,7 +8,6 @@ use runvil_parser::test_support::one_section; use super::*; -use crate::execute::tests::context::{setup_prompt, with_writer}; use crate::execute::tests::fixtures::echo_tool; /// The prompt `source` builds over the fixture tools Pack, run under the @@ -16,13 +15,13 @@ use crate::execute::tests::fixtures::echo_tool; /// `tools.call` runs is what `sys.model` reads after. #[track_caller] fn under_tools(source: PromptSource) -> Trace { - let source = setup_prompt(source.front("uses: [tools]")); + let source = source.front("uses: [tools]").build(); Scenario::new(source).bound().tool(echo_tool()).run() } #[test] fn a_library_function_sees_sys_model_unknown_before_scope_close() { - let source = with_writer(PromptSource::new()) + let source = PromptSource::new() .prose( "Only", " @@ -50,8 +49,7 @@ fn a_library_function_sees_sys_model_unknown_before_scope_close() { #[test] fn prologue_sys_model_unknown_before_scope_close() { - let source = - with_writer(PromptSource::new()).prose("Only", "```lua\nreturn sys.model\n```\n\nprose"); + let source = PromptSource::new().prose("Only", "```lua\nreturn sys.model\n```\n\nprose"); let trace = Scenario::new(source.build()).bound().run(); let error = trace.error(); assert!( @@ -127,7 +125,7 @@ fn fanout_arm_sees_the_model_name_in_sys_model_after_the_scope_install() { /// member reaches the model as compact JSON. #[test] fn fanout_item_substitution_renders_a_table_member_as_compact_json() { - let source = with_writer(PromptSource::new()) + let source = PromptSource::new() .section( "Parent", " diff --git a/crates/runvil-internal/engine/src/execute/tests/model_and_reply-handles.rs b/crates/runvil-internal/engine/src/execute/tests/model_and_reply-handles.rs index 85a26bf8c..ee68ac2c2 100644 --- a/crates/runvil-internal/engine/src/execute/tests/model_and_reply-handles.rs +++ b/crates/runvil-internal/engine/src/execute/tests/model_and_reply-handles.rs @@ -1,41 +1,31 @@ //! `models.get` and a handle's `infer`: a handle leaves the section //! model alone, `models.infer` uses the section model and never binds -//! `reply`, `models.use` re-selection steers the next round, and a -//! section with no current model fails `models.infer` but still infers -//! through a handle, and a handle's `provider` reads the catalog's. +//! `reply`, `models.use` re-selection steers the next round, a run whose +//! caller gave no model fails `models.infer`, and a handle's `provider` reads +//! the catalog's. use super::*; -use crate::execute::tests::context::{bind_role, with_writer}; +use crate::model::ModelRole; use crate::test_support::scenario::test_model; -/// The prompt declaring the `writer` and `analyst` roles and parking -/// `writer` as the prompt-wide default. -fn writer_and_analyst() -> PromptSource { - PromptSource::new() - .front("models: { writer: {}, analyst: {} }") - .h1("models.default('writer')") -} - -/// A run of `source` with the `writer` and `analyst` roles bound by hand to -/// `writer-model` and `analyst-model` and no prepare: the multi-model shape -/// the trivial fill cannot produce (it binds every role to the one current -/// model), exercising the runtime's label resolution directly. +/// A run of `source` whose caller maps the `default` role to `main-model` and +/// the `strong` role to `strong-model`: the two-model shape a one-model +/// caller cannot produce, exercising the runtime's role resolution. #[track_caller] -fn bound_by_hand(source: &PromptSource) -> Scenario { +fn two_models(source: &PromptSource) -> Scenario { Scenario::new(source.build()) - .unprepared() - .context(bind_role("writer", test_model("writer-model"))) - .context(bind_role("analyst", test_model("analyst-model"))) + .current_model(test_model("main-model")) + .role_model(ModelRole::Strong, test_model("strong-model")) } #[test] fn models_get_returns_a_handle_without_changing_the_section_model() { let gateway = ScriptedChat::new(vec![text("hello from the mock")]); - let source = writer_and_analyst().prose( + let source = PromptSource::new().prose( "Only", " ```lua - store.write('handle.txt', models.get('analyst').name) + store.write('handle.txt', models.get('strong').name) ``` Ask the model. @@ -44,20 +34,20 @@ fn models_get_returns_a_handle_without_changing_the_section_model() { return models.infer(prose) ```", ); - let trace = bound_by_hand(&source).chat(&gateway).run(); + let trace = two_models(&source).chat(&gateway).run(); assert_eq!(trace.text(), "hello from the mock"); assert_eq!( trace.read("handle.txt").unwrap(), - "analyst", - "models.get must return the analyst handle" + "strong", + "models.get must return the strong handle" ); let body = gateway .last_request() .expect("complete must reach the gateway"); assert_eq!( body.options.model(), - "writer-model", + "main-model", "models.get must not change the section's model" ); } @@ -65,7 +55,7 @@ fn models_get_returns_a_handle_without_changing_the_section_model() { #[test] fn models_infer_uses_the_section_model_without_touching_reply() { let gateway = ScriptedChat::new(vec![text("pong")]); - let source = with_writer(PromptSource::new()).prose( + let source = PromptSource::new().prose( "Only", " ```lua @@ -101,15 +91,16 @@ fn models_infer_uses_the_section_model_without_touching_reply() { #[test] fn handle_infer_uses_that_model_regardless_of_the_section_model() { let gateway = ScriptedChat::new(vec![text("pong")]); - let source = writer_and_analyst().section("Only", "return models.get('analyst'):infer('ping')"); - let trace = bound_by_hand(&source).chat(&gateway).run(); + let source = + PromptSource::new().section("Only", "return models.infer(models.get('strong'), 'ping')"); + let trace = two_models(&source).chat(&gateway).run(); assert_eq!(trace.text(), "pong"); let body = gateway .last_request() .expect("complete must reach the gateway"); assert_eq!( body.options.model(), - "analyst-model", + "strong-model", "a handle's infer must use the handle's model, not the section default" ); } @@ -117,15 +108,15 @@ fn handle_infer_uses_that_model_regardless_of_the_section_model() { #[test] fn models_use_reselection_steers_the_next_round() { let gateway = ScriptedChat::new(vec![text("first"), text("second")]); - let source = writer_and_analyst().section( + let source = PromptSource::new().section( "Only", " - models.use('writer') + models.use('default') models.infer('ping') - models.use('analyst') + models.use('strong') return models.infer('ping')", ); - let trace = bound_by_hand(&source).chat(&gateway).run(); + let trace = two_models(&source).chat(&gateway).run(); assert_eq!( trace.text(), "second", @@ -139,31 +130,20 @@ fn models_use_reselection_steers_the_next_round() { ); assert_eq!( requests[0].options.model(), - "writer-model", + "main-model", "the first round uses the initial selection" ); assert_eq!( requests[1].options.model(), - "analyst-model", + "strong-model", "the second round uses the re-selected model" ); } -/// The prompt declaring only the `analyst` role, with no prompt-wide -/// default, whose `Only` section runs `lua`. -fn analyst_only(lua: &str) -> String { - PromptSource::new() - .front("models: { analyst: {} }") - .section("Only", lua) - .build() -} - #[test] -fn models_infer_without_use_or_default_errors() { - let trace = Scenario::new(analyst_only("return models.infer('ping')")) - .unprepared() - .context(bind_role("analyst", test_model("analyst-model"))) - .run(); +fn models_infer_without_a_caller_model_errors() { + let source = PromptSource::new().section("Only", "return models.infer('ping')"); + let trace = Scenario::new(source.build()).run(); let error = trace.error(); assert!( error @@ -175,32 +155,14 @@ fn models_infer_without_use_or_default_errors() { #[test] fn a_handles_provider_is_the_catalog_provider_or_nil() { - let source = PromptSource::new() - .front("models: { writer: {}, analyst: {} }") - .section( - "Only", - "return tostring(models.get('writer').provider) .. ':' .. tostring(models.get('analyst').provider)", - ); - let writer = test_model("writer-model").with_provider("xai"); + let source = PromptSource::new().section( + "Only", + "return tostring(models.get('default').provider) .. ':' .. tostring(models.get('strong').provider)", + ); + let main = test_model("main-model").with_provider("xai"); let trace = Scenario::new(source.build()) - .unprepared() - .context(bind_role("writer", writer)) - .context(bind_role("analyst", test_model("analyst-model"))) + .current_model(main) + .role_model(ModelRole::Strong, test_model("strong-model")) .run(); assert_eq!(trace.text(), "xai:nil"); } - -#[test] -fn models_get_infer_works_without_any_section_model() { - let gateway = ScriptedChat::new(vec![text("pong")]); - let trace = Scenario::new(analyst_only("return models.get('analyst'):infer('ping')")) - .unprepared() - .context(bind_role("analyst", test_model("analyst-model"))) - .chat(&gateway) - .run(); - assert_eq!(trace.text(), "pong"); - let body = gateway - .last_request() - .expect("complete must reach the gateway"); - assert_eq!(body.options.model(), "analyst-model"); -} diff --git a/crates/runvil-internal/engine/src/execute/tests/model_and_reply.rs b/crates/runvil-internal/engine/src/execute/tests/model_and_reply.rs index 18e2c9525..0b0c60090 100644 --- a/crates/runvil-internal/engine/src/execute/tests/model_and_reply.rs +++ b/crates/runvil-internal/engine/src/execute/tests/model_and_reply.rs @@ -4,9 +4,9 @@ use runvil_parser::test_support::{PromptSource, parse_ok}; +use crate::model::ModelRole; use crate::test_support::scenario::{Trace, test_model}; -use super::context::{bind_role, with_writer}; use super::*; /// The run's observations as section and detail pairs, from its start on: @@ -23,19 +23,19 @@ fn run_events(trace: &Trace) -> Vec<(String, String)> { #[test] fn models_use_forwards_binding_completion_options_to_the_gateway() { // models.use -> completion_options -> GatewayChat::complete must set - // the binding's model, the hard-keyword thinking switch, and the + // the binding's model, the thinking switch, and the // section's `models.use` sampling options on the chat body. Roles - // declare no sampling fields, so a section on the prompt-wide default - // sends none. + // declare no sampling fields, so a round on the `default` role sends + // none, and the second section switches back to it explicitly because + // the first section's switch lasts across the chain. let gateway = ScriptedChat::new(vec![text("hello from the mock"), text("hello again")]); let source = PromptSource::new() - .front("models: { analyst: { keywords: [no-thinking] } }") - .h1("models.default('analyst')") + .front("models: { small: { thinking: false } }") .prose( "Only", " ```lua - models.use('analyst', { temperature = 0, max_tokens = 256 }) + models.use('small', { temperature = 0, max_tokens = 256 }) ``` Ask the model. @@ -50,12 +50,13 @@ fn models_use_forwards_binding_completion_options_to_the_gateway() { Ask again. ```lua + models.use('default') return models.infer(prose) ```", ); let trace = Scenario::new(source.build()) - .unprepared() - .context(bind_role("analyst", test_model("analyst"))) + .current_model(test_model("main-model")) + .role_model(ModelRole::Small, test_model("small-model")) .chat(&gateway) .run(); assert_eq!(trace.text(), "hello again"); @@ -63,7 +64,7 @@ fn models_use_forwards_binding_completion_options_to_the_gateway() { let requests = gateway.requests(); assert_eq!(requests.len(), 2, "one round per section: {requests:?}"); let selected = &requests[0]; - assert_eq!(selected.options.model(), "analyst"); + assert_eq!(selected.options.model(), "small-model"); assert_eq!(selected.options.thinking(), Some(false)); assert_eq!( selected @@ -77,20 +78,18 @@ fn models_use_forwards_binding_completion_options_to_the_gateway() { Some(256) ); let defaulted = &requests[1]; - assert_eq!(defaulted.options.model(), "analyst"); - assert_eq!(defaulted.options.thinking(), Some(false)); + assert_eq!(defaulted.options.model(), "main-model"); + assert_eq!(defaulted.options.thinking(), None); assert!( defaulted.options.temperature().is_none() && defaulted.options.max_tokens().is_none(), - "a section on the prompt-wide default sends neither option: {defaulted:?}" + "a round on the `default` role sends neither option: {defaulted:?}" ); } /// One section that infers its prose, `Say something.`, then runs `epilog`. fn say_something(epilog: &str) -> String { let body = format!("Say something.\n\n```lua\n{epilog}\n```"); - with_writer(PromptSource::new()) - .prose("Only", &body) - .build() + PromptSource::new().prose("Only", &body).build() } #[test] @@ -109,22 +108,6 @@ fn an_explicit_client_is_used_instead_of_the_environment() { run_events(&trace), vec![ ("Test prompt".to_string(), detail::RUN_STARTED.to_string()), - ( - "Test prompt".to_string(), - detail::LUA_CHUNK_STARTED.to_string(), - ), - ( - "Test prompt".to_string(), - detail::LUA_CHUNK_SUCCEEDED.to_string(), - ), - ( - "Test prompt".to_string(), - detail::LUA_TEARDOWN_STARTED.to_string(), - ), - ( - "Test prompt".to_string(), - detail::LUA_TEARDOWN_SUCCEEDED.to_string(), - ), ("Only".to_string(), detail::SECTION_STARTED.to_string()), ("Only".to_string(), detail::LUA_CHUNK_STARTED.to_string()), ("Only".to_string(), detail::MODEL_TURN_COMPLETED.to_string(),), @@ -162,22 +145,6 @@ fn epilog_runs_after_prose_and_can_return() { run_events(&trace), vec![ ("Test prompt".to_string(), detail::RUN_STARTED.to_string()), - ( - "Test prompt".to_string(), - detail::LUA_CHUNK_STARTED.to_string(), - ), - ( - "Test prompt".to_string(), - detail::LUA_CHUNK_SUCCEEDED.to_string(), - ), - ( - "Test prompt".to_string(), - detail::LUA_TEARDOWN_STARTED.to_string(), - ), - ( - "Test prompt".to_string(), - detail::LUA_TEARDOWN_SUCCEEDED.to_string(), - ), ("Only".to_string(), detail::SECTION_STARTED.to_string()), ("Only".to_string(), detail::LUA_CHUNK_STARTED.to_string()), ("Only".to_string(), detail::MODEL_TURN_COMPLETED.to_string()), @@ -260,7 +227,7 @@ fn prologue_return_skips_model_and_epilog() { #[test] fn a_required_helper_survives_prologue_model_and_epilog() { let gateway = ScriptedChat::new(vec![text("hello from the mock")]); - let source = with_writer(PromptSource::new()) + let source = PromptSource::new() .prose( "Only", " @@ -290,22 +257,6 @@ fn a_required_helper_survives_prologue_model_and_epilog() { run_events(&trace), [ ("Test prompt".to_owned(), detail::RUN_STARTED.to_string()), - ( - "Test prompt".to_owned(), - detail::LUA_CHUNK_STARTED.to_string(), - ), - ( - "Test prompt".to_owned(), - detail::LUA_CHUNK_SUCCEEDED.to_string(), - ), - ( - "Test prompt".to_owned(), - detail::LUA_TEARDOWN_STARTED.to_string(), - ), - ( - "Test prompt".to_owned(), - detail::LUA_TEARDOWN_SUCCEEDED.to_string(), - ), ("Only".to_owned(), detail::SECTION_STARTED.to_string()), ("Only".to_owned(), detail::LUA_CHUNK_STARTED.to_string()), ( @@ -380,7 +331,7 @@ fn model_required_when_infer_has_no_binding() { let error = trace.error(); assert_eq!( error.kind(), - RunErrorKind::Binding, + RunErrorKind::RequirementsUnmet, "expected ModelRequired, got {error}" ); assert!( diff --git a/crates/runvil-internal/engine/src/execute/tests/models_loop-arguments.rs b/crates/runvil-internal/engine/src/execute/tests/models_loop-arguments.rs index d5f4e4e65..1466a42ae 100644 --- a/crates/runvil-internal/engine/src/execute/tests/models_loop-arguments.rs +++ b/crates/runvil-internal/engine/src/execute/tests/models_loop-arguments.rs @@ -1,7 +1,6 @@ -//! Tests for how `models.loop` and a model handle's `loop` read their -//! arguments: every documented form runs over the section's `messages` -//! list, and each entry's argument errors are raised in order before any -//! request leaves. +//! Tests for how `models.loop` reads its arguments: every documented form +//! runs over the section's `messages` list, and the argument errors are +//! raised in order before any request leaves. use runvil_parser::test_support::one_section; @@ -16,7 +15,7 @@ use super::*; fn refusals(calls: &str) -> String { let gateway = ScriptedChat::new(vec![text("unreachable")]); let source = one_section(&format!( - "local other = models.get('other') + "local other = models.get('strong') messages:user('hello') local out = {{}} for _, call in ipairs({{ {calls} }}) do @@ -39,15 +38,15 @@ fn refusals(calls: &str) -> String { fn every_argument_form_runs_over_the_sections_list() { let gateway = ScriptedChat::new(vec![text("r1"), text("r2"), text("r3"), text("r4")]); let source = one_section( - "local other = models.get('other') + "local other = models.get('strong') messages:user('one') models.loop() messages:user('two') models.loop(compactors.fail) messages:user('three') - other:loop() + models.loop(other) messages:user('four') - other:loop(compactors.fail) + models.loop(other, compactors.fail) local replies = {} for i = 2, #messages, 2 do replies[#replies + 1] = messages[i].content end return #messages .. '|' .. table.concat(replies, ',')", @@ -62,7 +61,7 @@ fn every_argument_form_runs_over_the_sections_list() { assert_eq!( models, ["test-model", "test-model", "other-model", "other-model"], - "only the handle's own loop runs on the handle's binding" + "a leading handle runs on the handle's binding, and its absence on the section's model" ); let sizes: Vec = gateway .requests() @@ -77,62 +76,33 @@ fn every_argument_form_runs_over_the_sections_list() { } #[test] -fn a_handle_passed_to_models_loop_is_the_pointed_error() { - let out = refusals( - "function() return models.loop(other) end, - function() return models.loop(other, compactors.fail) end,", - ); - let pointed = "lua:models.loop takes (compactor?); \ - call handle:loop(compactor?) to run on a model handle"; - assert_eq!(out, format!("{pointed}|{pointed}|1")); -} - -#[test] -fn a_handles_loop_called_without_its_receiver_is_the_colon_error() { - // A dot call passes the compactor where the receiver belongs, and the - // function taken off the handle and called bare has no receiver at all. - let out = refusals( - "function() return other.loop(compactors.fail) end, - function() local run = other.loop; return run(compactors.fail) end, - function() return other.loop() end,", - ); - let colon = "lua:call loop on a model handle with a colon: handle:loop(compactor?)"; - assert_eq!(out, format!("{colon}|{colon}|{colon}|1")); -} - -#[test] -fn each_loop_entry_refuses_extra_arguments_counting_trailing_nils() { +fn the_arity_error_counts_trailing_nils_in_both_shapes() { let out = refusals( "function() return models.loop(compactors.fail, 1) end, function() return models.loop(nil, nil) end, - function() return other:loop(compactors.fail, 1) end, - function() return other:loop(nil, nil) end,", + function() return models.loop(other, compactors.fail, 1) end, + function() return models.loop(other, nil, nil) end, + function() return models.loop(other, compactors.fail, nil) end,", ); - let entry = "lua:models.loop takes (compactor?)"; - let method = "lua:handle:loop takes (compactor?)"; - assert_eq!(out, format!("{entry}|{entry}|{method}|{method}|1")); + let arity = "lua:models.loop takes (compactor?) or (model, compactor?)"; + assert_eq!(out, format!("{arity}|{arity}|{arity}|{arity}|{arity}|1")); } #[test] -fn each_entry_checks_the_handle_then_the_arity_then_the_compactor() { - // Every call but the last of each entry fails two adjacent checks, so - // the earlier check names the error. +fn the_compactor_is_checked_after_the_arity() { + // Every call but the last fails two adjacent checks, so the earlier + // check names the error. let out = refusals( - "function() return models.loop(other, 1) end, - function() return models.loop(42, nil) end, + "function() return models.loop(42, nil) end, function() return models.loop(42) end, - function() return other.loop(nil, nil, nil) end, - function() return other:loop(42, nil) end, - function() return other:loop(42) end,", + function() return models.loop(other, 42, nil) end, + function() return models.loop(other, 42) end,", ); assert_eq!( out, - "lua:models.loop takes (compactor?); \ - call handle:loop(compactor?) to run on a model handle\ - |lua:models.loop takes (compactor?)\ + "lua:models.loop takes (compactor?) or (model, compactor?)\ |lua:compactor must be a function, got integer\ - |lua:call loop on a model handle with a colon: handle:loop(compactor?)\ - |lua:handle:loop takes (compactor?)\ + |lua:models.loop takes (compactor?) or (model, compactor?)\ |lua:compactor must be a function, got integer\ |1" ); diff --git a/crates/runvil-internal/engine/src/execute/tests/models_loop-author-shapes.rs b/crates/runvil-internal/engine/src/execute/tests/models_loop-author-shapes.rs index dd759445f..2f7c95479 100644 --- a/crates/runvil-internal/engine/src/execute/tests/models_loop-author-shapes.rs +++ b/crates/runvil-internal/engine/src/execute/tests/models_loop-author-shapes.rs @@ -228,13 +228,12 @@ fn the_chat_agent_shape_leaves_the_oldest_records_out_of_a_summary_past_its_budg )); let mut wide = ModelSet::default(); wide.push(ModelBinding::new( - "writer", + "default", "A general model for tests", ModelId::gateway("test-model").expect("the test model id is valid"), ModelInvocation::new(), NonZeroU32::new(131_072).expect("the window is non-zero"), )); - wide.set_default_alias("writer".to_owned()); let trace = loop_scenario(source).models(wide).chat(&gateway).run(); assert_eq!(trace.text(), "answer|true"); let bodies = gateway.requests(); diff --git a/crates/runvil-internal/engine/src/execute/tests/models_loop.rs b/crates/runvil-internal/engine/src/execute/tests/models_loop.rs index 6ea59ca6b..8c53e5f80 100644 --- a/crates/runvil-internal/engine/src/execute/tests/models_loop.rs +++ b/crates/runvil-internal/engine/src/execute/tests/models_loop.rs @@ -6,7 +6,7 @@ //! round renders, explicit-handle calls on a //! frozen binding, and the atomic append of a tool-call batch. The shared //! loop fixtures (`loop_models`, `loop_model_name`, `loop_scenario`, -//! `catalog_scenario`, `LOOP_TO_TEXT`, and the `loop_events` filter) sit +//! `loop_scenario`, `LOOP_TO_TEXT`, and the `loop_events` filter) sit //! here for every loop-driven sibling. //! The compactor argument's tests sit in `models_loop_compactors`; the //! loop's exit rules in `exit_rules`; its cap, scope gate, and @@ -25,18 +25,17 @@ mod arguments; #[path = "models_loop-author-shapes.rs"] mod author_shapes; -/// The model set a loop test's run uses: `writer` (the prompt-wide -/// default, model `test-model`) and `other` (model `other-model`), so an +/// The model set a loop test's run uses: `default` (the role every section +/// starts on, model `test-model`) and `strong` (model `other-model`), so an /// explicit handle provably runs on its own frozen binding. pub(super) fn loop_models() -> ModelSet { let mut set = ModelSet::default(); set.push(test_binding( - "writer", + "default", "A general model for tests", "test-model", )); - set.push(test_binding("other", "A second model", "other-model")); - set.set_default_alias("writer".to_owned()); + set.push(test_binding("strong", "A second model", "other-model")); set } @@ -53,15 +52,9 @@ pub(super) fn loop_model_name(alias: &str) -> String { .to_owned() } -/// A run of `source` over the loop models, unprepared, so the run starts -/// with what the H1 pass would leave and no catalog. +/// A run of `source` over the loop models, so the run starts with what the +/// H1 pass would leave. pub(super) fn loop_scenario(source: impl Into) -> Scenario { - Scenario::new(source).unprepared().models(loop_models()) -} - -/// A run of `source` over the loop models, prepared, so the scenario's -/// catalog tools are offered as the prepare offers them. -pub(super) fn catalog_scenario(source: impl Into) -> Scenario { Scenario::new(source).models(loop_models()) } @@ -281,11 +274,11 @@ fn an_offer_after_the_first_loop_raises_and_the_second_loop_sends_the_same_empty fn a_handles_loop_runs_on_its_frozen_binding() { let gateway = ScriptedChat::new(vec![text("first"), text("second")]); let source = one_section( - "local other = models.get('other') + "local other = models.get('strong') messages:user('one') - other:loop() + models.loop(other) messages:user('two') - other:loop() + models.loop(other) assert(#messages == 4, 'each call appends its terminal record') return messages[2].content .. '|' .. messages[4].content", ); diff --git a/crates/runvil-internal/engine/src/execute/tests/models_loop_compactors.rs b/crates/runvil-internal/engine/src/execute/tests/models_loop_compactors.rs index 866877bab..cbe8e92ca 100644 --- a/crates/runvil-internal/engine/src/execute/tests/models_loop_compactors.rs +++ b/crates/runvil-internal/engine/src/execute/tests/models_loop_compactors.rs @@ -24,7 +24,7 @@ fn an_omitted_compactor_defaults_to_fail_with_typed_precheck_exhaustion() { return 'unreachable'", ); let trace = loop_scenario(source).chat(&gateway).run(); - let error = trace.error().inner(); + let error = trace.error(); assert!( matches!(error, Error::ContextExhausted { reason, .. } if reason == "precheck"), "the omitted compactor defaults to compactors.fail with the precheck reason, got {error:?}" @@ -67,7 +67,7 @@ fn an_explicit_compactors_fail_invocation_reports_the_provider_reason() { return 'unreachable'", ); let trace = loop_scenario(source).chat(&gateway).run(); - let error = trace.error().inner(); + let error = trace.error(); assert!( matches!(error, Error::ContextExhausted { reason, .. } if reason == "provider"), "the explicit compactors.fail invocation reports the provider reason, got {error:?}" @@ -91,9 +91,8 @@ fn an_uncaught_compactors_fail_with_a_callers_reason_ends_the_run_as_context_exh "the run's message holds the caller's reason, got: {error}" ); assert!( - matches!(error.inner(), Error::ContextExhausted { reason, .. } if reason == "manual"), - "the reason survives as given, got {:?}", - error.inner() + matches!(error, Error::ContextExhausted { reason, .. } if reason == "manual"), + "the reason survives as given, got {error:?}" ); assert_eq!( gateway.call_count(), @@ -117,9 +116,8 @@ fn an_uncaught_direct_compactors_fail_ends_the_run_as_context_exhausted() { "the run's message holds the caller's reason, got: {error}" ); assert!( - matches!(error.inner(), Error::ContextExhausted { reason, .. } if reason == "manual"), - "the reason survives as given, got {:?}", - error.inner() + matches!(error, Error::ContextExhausted { reason, .. } if reason == "manual"), + "the reason survives as given, got {error:?}" ); assert_eq!(gateway.call_count(), 0, "no request leaves"); } @@ -129,12 +127,12 @@ fn a_non_function_compactor_is_the_calls_error_in_the_engines_type_names() { // The argument error is pcall-able at the call site and names the // value's type as the protocol parse does: an integer is "integer", // a float "number", anything else its Lua type name. No round runs. - // Each value is passed to `models.loop` and to a handle's `loop`; - // both entries reach the compactor check. + // Each value is passed to `models.loop` with and without a leading + // handle; both shapes reach the compactor check. let gateway = ScriptedChat::new(vec![text("unreachable")]); let source = one_section( "messages:user('hello') - local writer = models.get('writer') + local writer = models.get('default') local function refusal(entry, ...) local ok, err = pcall(entry, ...) assert(not ok, 'a non-function compactor raises') @@ -144,7 +142,7 @@ fn a_non_function_compactor_is_the_calls_error_in_the_engines_type_names() { local out = {} for _, bad in ipairs({ 42, 4.5, 'summarize', {} }) do out[#out + 1] = refusal(models.loop, bad) - out[#out + 1] = refusal(writer.loop, writer, bad) + out[#out + 1] = refusal(models.loop, writer, bad) end assert(#messages == 1, 'a refused call appends nothing') return table.concat(out, '|')", @@ -213,9 +211,9 @@ fn cancel_during_a_looping_compactor_returns_promptly() { .cancel_when(|effect| matches!(effect, Effect::Chat { .. })) .run(); assert!( - matches!(trace.result(), RunResult::Cancelled), + matches!(trace.outcome(), RunOutcome::Cancelled), "expected the cancelled run, got {:?}", - trace.result() + trace.outcome() ); assert_eq!( gateway.call_count(), @@ -237,7 +235,7 @@ fn a_compactors_own_string_raise_reaches_the_caller_with_the_reason_tag() { return 'unreachable'", ); let trace = loop_scenario(source).chat_with(overflowing(&rounds)).run(); - match trace.error().inner() { + match trace.error() { Error::Lua { message, cause: Some(_), diff --git a/crates/runvil-internal/engine/src/execute/tests/models_loop_contract-rounds.rs b/crates/runvil-internal/engine/src/execute/tests/models_loop_contract-rounds.rs index 8058d2170..e336b0867 100644 --- a/crates/runvil-internal/engine/src/execute/tests/models_loop_contract-rounds.rs +++ b/crates/runvil-internal/engine/src/execute/tests/models_loop_contract-rounds.rs @@ -222,7 +222,7 @@ fn a_dropped_model_issued_call_raises_cancelled_at_the_loop_call_site() { ); run.resume(call, EffectAnswer::Dropped); let result = match run.step() { - Step::Done { result, .. } => result, + Step::Done { outcome, .. } => outcome, other @ Step::Pending { .. } => panic!("the caught drop ends the section: {other:?}"), }; assert_eq!(text_of(result), "false:cancelled|1|user"); @@ -274,9 +274,9 @@ fn an_adhoc_tool_cancelled_mid_handler_reports_no_failed_tool_call() { driving.cancel(); let trace = driving.finish(); assert!( - matches!(trace.result(), RunResult::Cancelled), + matches!(trace.outcome(), RunOutcome::Cancelled), "expected the cancelled run, got {:?}", - trace.result() + trace.outcome() ); assert_eq!( trace_loop_events(&trace), diff --git a/crates/runvil-internal/engine/src/execute/tests/models_loop_contract.rs b/crates/runvil-internal/engine/src/execute/tests/models_loop_contract.rs index 87c789e14..36f080037 100644 --- a/crates/runvil-internal/engine/src/execute/tests/models_loop_contract.rs +++ b/crates/runvil-internal/engine/src/execute/tests/models_loop_contract.rs @@ -75,9 +75,9 @@ fn a_light_userdata_compactor_is_named_userdata_as_lua_names_it() { let source = one_section( "tools.offer('tools/structured') local null = tools.call('tools/structured', {}).none - local other = models.get('other') + local other = models.get('strong') messages:user('hello') - local ok, err = pcall(other.loop, other, null) + local ok, err = pcall(models.loop, other, null) assert(not ok, 'a light userdata compactor is refused') return type(null) .. '|' .. err.kind .. '|' .. tostring(err) .. '|' .. #messages", ); @@ -113,11 +113,14 @@ fn the_arity_error_still_raises_when_the_author_has_rebound_error() { driving.cancel(); let trace = driving.finish(); assert!( - !matches!(trace.result(), RunResult::Cancelled), + !matches!(trace.outcome(), RunOutcome::Cancelled), "the raise ends the call before the cancel: {:?}", - trace.result() + trace.outcome() + ); + assert_eq!( + trace.text(), + "models.loop takes (compactor?) or (model, compactor?)" ); - assert_eq!(trace.text(), "models.loop takes (compactor?)"); assert!(trace.effects().is_empty(), "no round runs"); } @@ -144,7 +147,7 @@ fn an_uncaught_loop_arity_error_fails_the_run_naming_the_authors_prompt_line() { return 'unreachable'", ); let trace = loop_scenario(source).chat(&gateway).run(); - match trace.error().inner() { + match trace.error() { Error::Lua { message, cause: Some(_), @@ -170,8 +173,8 @@ fn each_loop_entry_returns_exactly_one_value() { "messages:user('first') local n = select('#', models.loop()) messages:user('second') - local other = models.get('other') - local m = select('#', other:loop()) + local other = models.get('strong') + local m = select('#', models.loop(other)) return n .. '|' .. m .. '|' .. #messages", ); let trace = loop_scenario(source).chat(&gateway).run(); diff --git a/crates/runvil-internal/engine/src/execute/tests/observations.rs b/crates/runvil-internal/engine/src/execute/tests/observations.rs index 8e5f81c10..ba2c8eff8 100644 --- a/crates/runvil-internal/engine/src/execute/tests/observations.rs +++ b/crates/runvil-internal/engine/src/execute/tests/observations.rs @@ -160,9 +160,9 @@ fn a_run_refused_by_the_version_gate_reports_nothing() { let source = one_section("return \"ran\"").replacen("runvil: 0", "runvil: 2", 1); let trace = Scenario::new(source).run(); assert!( - matches!(trace.result(), RunResult::Failure(_)), + matches!(trace.outcome(), RunOutcome::Failed { .. }), "the version gate fails the run, got {:?}", - trace.result() + trace.outcome() ); let records = trace.run_records(); assert!( @@ -177,7 +177,7 @@ fn a_failing_run_still_reports_run_finished() { // observation must report the run failure. let trace = Scenario::new(failing_prompt()).run(); assert!( - matches!(trace.error().inner(), Error::Lua { .. }), + matches!(trace.error(), Error::Lua { .. }), "the prologue fails as a Lua error, got {:?}", trace.error() ); @@ -206,9 +206,9 @@ fn an_erroring_section_reports_started_but_not_finished() { // errors mid-walk must emit SECTION_STARTED and never SECTION_FINISHED. let trace = Scenario::new(failing_prompt()).run(); assert!( - matches!(trace.result(), RunResult::Failure(_)), + matches!(trace.outcome(), RunOutcome::Failed { .. }), "the erroring section fails the run, got {:?}", - trace.result() + trace.outcome() ); let observed = sequence(&trace); @@ -231,9 +231,9 @@ fn an_erroring_section_tears_down_exactly_once_without_finishing() { // keeps SECTION_FINISHED from firing for the erroring section. let trace = Scenario::new(second_section_errors()).run(); assert!( - matches!(trace.result(), RunResult::Failure(_)), + matches!(trace.outcome(), RunOutcome::Failed { .. }), "the erroring section fails the run, got {:?}", - trace.result() + trace.outcome() ); let observed = sequence(&trace); @@ -289,9 +289,8 @@ fn a_one_byte_limit_fails_value_injection_with_teardown_observations() { fn one_execution_id_spans_parse_and_the_complete_runtime_lifecycle() { let gateway = ScriptedChat::new(vec![text("aliased final")]); let (tool, called) = scoped_tool("echo", "Echo a test value."); - let source = "---\nname: lifecycle\ndescription: Correlated lifecycle fixture\nrunvil: 0\nuses:\n - tools\nmodels:\n writer: {}\n---\n\n\ - # Lifecycle\n\n```lua\n\ - models.default('writer')\n```\n\n\ + let source = "---\nname: lifecycle\ndescription: Correlated lifecycle fixture\nrunvil: 0\nuses:\n - tools\n---\n\n\ + # Lifecycle\n\n\ ## Gather\n\n```lua\ntools.offer('tools/echo')\nstore.write('state.txt', 'before')\n```\n\n\ Use the echo tool.\n\n\ ```lua\n\ diff --git a/crates/runvil-internal/engine/src/execute/tests/offering-packs.rs b/crates/runvil-internal/engine/src/execute/tests/offering-packs.rs index 9ba46ce26..4f5dffdf8 100644 --- a/crates/runvil-internal/engine/src/execute/tests/offering-packs.rs +++ b/crates/runvil-internal/engine/src/execute/tests/offering-packs.rs @@ -52,7 +52,7 @@ fn an_offered_pack_reaches_only_the_rounds_of_the_section_entry_that_offered_it( ); let gateway = ScriptedChat::new(vec![text("done"); 5]); let rig = Rig::run( - catalog_scenario(source) + loop_scenario(source) .tool(offered_tool("kit/a")) .tool(offered_tool("kit/b")) .tool(offered_tool("web/fetch")) @@ -110,7 +110,7 @@ fn two_sections_that_offer_one_pack_each_route_a_model_call_by_their_own_sets() ], ); let rig = Rig::run( - catalog_scenario(source) + loop_scenario(source) .tool(offered_tool("kit/a")) .tool(offered_tool("kit/b")) .chat(&gateway), diff --git a/crates/runvil-internal/engine/src/execute/tests/offering-sets.rs b/crates/runvil-internal/engine/src/execute/tests/offering-sets.rs index 82f0cf331..8b7f42ce6 100644 --- a/crates/runvil-internal/engine/src/execute/tests/offering-sets.rs +++ b/crates/runvil-internal/engine/src/execute/tests/offering-sets.rs @@ -45,7 +45,7 @@ fn two_catalog_tools_whose_default_names_clash_fail_only_the_section_that_offers ], ); let rig = Rig::run( - catalog_scenario(source) + loop_scenario(source) .tool(offered_tool("dup/a.b")) .tool(offered_tool("dup/a/b")) .chat(&gateway), @@ -83,7 +83,7 @@ fn a_catalog_tool_whose_default_name_is_invalid_fails_only_the_sections_that_off ], ); let rig = Rig::run( - catalog_scenario(source) + loop_scenario(source) .tool(echo_tool()) .tool(offered_tool("9lives/fetch")) .chat(&gateway), @@ -111,7 +111,7 @@ fn a_catalog_tool_is_advertised_and_called_under_its_own_wire_name() { return messages[3].content", ); let rig = Rig::run( - catalog_scenario(source) + loop_scenario(source) .tool(echo_tool().wire_name("echo")) .chat(&gateway), ); @@ -128,7 +128,7 @@ fn the_offer_hint_names_a_catalog_tool_by_its_own_wire_name_only() { local _, default = pcall(tools.offer, 'tools_echo')\n\ return tostring(own) .. '|' .. tostring(default)", ); - let rig = Rig::run(catalog_scenario(source).tool(echo_tool().wire_name("echo"))); + let rig = Rig::run(loop_scenario(source).tool(echo_tool().wire_name("echo"))); let (own, default) = rig .trace() .text() @@ -149,12 +149,12 @@ fn the_scope_gate_says_a_name_exists_only_for_a_catalog_tools_own_wire_name() { let refused = |name: &str| { let gateway = ScriptedChat::new(vec![calls(&[("call_1", name, "{}")])]); let source = one_section("tools.offer('gh/search')\nmessages:user('go')\nmodels.loop()"); - let trace = catalog_scenario(source) + let trace = loop_scenario(source) .tool(echo_tool().wire_name("echo")) .tool(offered_tool("gh/search")) .chat(&gateway) .run(); - match trace.error().inner() { + match trace.error() { Error::OutOfScopeToolCall { global_exists, has_dynamic_set, @@ -188,7 +188,7 @@ fn the_scope_validation_events_bracket_the_merge_whether_it_holds_or_fails() { ("Clash", &failing_round("dup")), ], ); - let trace = catalog_scenario(source) + let trace = loop_scenario(source) .tool(offered_tool("dup/a.b")) .tool(offered_tool("dup/a/b")) .chat(&gateway) @@ -231,7 +231,7 @@ fn a_section_that_interleaves_two_packs_offers_sees_each_packs_tools_together() models.loop()", ); let rig = Rig::run( - catalog_scenario(source) + loop_scenario(source) .tool(offered_tool("kit/a")) .tool(offered_tool("kit/b")) .tool(offered_tool("web/fetch")) diff --git a/crates/runvil-internal/engine/src/execute/tests/offering.rs b/crates/runvil-internal/engine/src/execute/tests/offering.rs index b8849f48b..5c8416104 100644 --- a/crates/runvil-internal/engine/src/execute/tests/offering.rs +++ b/crates/runvil-internal/engine/src/execute/tests/offering.rs @@ -5,8 +5,8 @@ use runvil_parser::test_support::{PromptSource, one_section, prompt}; -use super::fixtures::echo_tool; -use super::models_loop::{catalog_scenario, loop_model_name}; +use super::fixtures::{echo_tool, model_only_tool}; +use super::models_loop::{loop_model_name, loop_scenario}; use super::*; use crate::execute::run::{EffectRecord, ToolCallOrigin, ToolCaller}; use crate::test_support::rig::Rig; @@ -51,7 +51,7 @@ fn the_run_catalog_keeps_its_order_and_a_pack_offer_expands_in_id_order() { )], ); let mut rig = Rig::start( - catalog_scenario(source) + loop_scenario(source) .tool(echo_tool()) .tool(offered_tool("gh/search")) .tool(offered_tool("alpha/zed")) @@ -109,7 +109,7 @@ fn a_script_offers_the_extras_and_the_model_calls_an_offered_tool_by_its_wire_na models.loop()\n\ return 'ok'", ); - let rig = Rig::run(catalog_scenario(source).tool(echo_tool()).chat(&gateway)); + let rig = Rig::run(loop_scenario(source).tool(echo_tool()).chat(&gateway)); assert_eq!(rig.trace().text(), "ok"); let records = rig.issued(); let advertised: Vec> = records @@ -135,6 +135,36 @@ fn a_script_offers_the_extras_and_the_model_calls_an_offered_tool_by_its_wire_na ); } +#[test] +fn a_model_only_tool_is_advertised_and_called_by_the_model() { + // The flag gates a script's call alone: a section offers the tool and + // a model round calls it by its wire name, which is the whole point of + // keeping the tool in the catalog. + let gateway = ScriptedChat::new(vec![ + calls(&[("call_1", "tools_replace", r#"{"value":"hi"}"#)]), + text("done"), + ]); + let source = one_section( + "tools.offer('tools/replace')\n\ + messages:user('go')\n\ + models.loop()\n\ + return 'ok'", + ); + let rig = Rig::run(loop_scenario(source).tool(model_only_tool()).chat(&gateway)); + assert_eq!(rig.trace().text(), "ok"); + let records = rig.issued(); + let calls: Vec<&EffectRecord> = records + .iter() + .filter(|record| matches!(record, EffectRecord::ToolCall { .. })) + .collect(); + assert_eq!(calls.len(), 1, "the model's call reached the tool"); + assert!(matches!( + calls[0], + EffectRecord::ToolCall { tool, alias, .. } + if tool.to_string() == "tools/replace" && alias == "tools_replace" + )); +} + /// Every key on a section's `tools` table, sorted, each with its Lua type. const TOOLS_TABLE_KEYS: &str = "adhoc:function,call:function,get:function,offer:function,offer_dynamic:function,packs:table"; @@ -151,7 +181,7 @@ fn a_section_tools_table_holds_only_its_api_and_its_round_advertises_only_the_of models.loop()\n\ return table.concat(keys, ',')", ); - let rig = Rig::run(catalog_scenario(source).tool(echo_tool()).chat(&gateway)); + let rig = Rig::run(loop_scenario(source).tool(echo_tool()).chat(&gateway)); assert_eq!( rig.trace().text(), TOOLS_TABLE_KEYS, @@ -179,7 +209,7 @@ fn a_script_calls_an_offered_tool_by_its_object_and_by_its_id_without_offering_i return tools.call(tool, { value = 'a' }) .. '|' .. \ tools.call('tools/echo', { value = 'b' }) .. '|' .. tool.id", ); - let rig = Rig::run(catalog_scenario(source).tool(echo_tool())); + let rig = Rig::run(loop_scenario(source).tool(echo_tool())); assert_eq!(rig.trace().text(), "echoed: a|echoed: b|tools/echo"); let call = |value: &str| EffectRecord::ToolCall { tool: ToolId::parse("tools/echo").expect("valid id"), @@ -213,7 +243,7 @@ fn rounds_and_calls(records: &[EffectRecord]) -> (Vec>, Vec ToolCallOrigin { ToolCallOrigin::new(EXECUTION, section, ToolCaller::Model) .with_entry("0.1") - .with_model(loop_model_name("writer")) + .with_model(loop_model_name("default")) } /// A model call of `id` under `alias` with `value`, from the one section. @@ -243,7 +273,7 @@ fn a_script_offers_a_declared_packs_tools_and_the_model_calls_one_by_its_wire_na )], ); let rig = Rig::run( - catalog_scenario(source) + loop_scenario(source) .tool(echo_tool()) .tool(offered_tool("gh/search")) .chat(&gateway), @@ -273,7 +303,7 @@ fn a_script_offers_an_extra_packs_tools_after_checking_packs_get() { return 'ok'", ); let rig = Rig::run( - catalog_scenario(source) + loop_scenario(source) .tool(echo_tool()) .tool(offered_tool("gh/search")) .chat(&gateway), @@ -300,7 +330,7 @@ fn the_pack_objects_and_the_adhoc_pack_are_installed_in_the_h1_vm() { ) .build(); let rig = Rig::run( - catalog_scenario(source) + loop_scenario(source) .tool(echo_tool()) .tool(offered_tool("gh/search")), ); diff --git a/crates/runvil-internal/engine/src/execute/tests/parallel_tool_calls-withdrawn.rs b/crates/runvil-internal/engine/src/execute/tests/parallel_tool_calls-withdrawn.rs index 3eec61c5b..21c7dbdc5 100644 --- a/crates/runvil-internal/engine/src/execute/tests/parallel_tool_calls-withdrawn.rs +++ b/crates/runvil-internal/engine/src/execute/tests/parallel_tool_calls-withdrawn.rs @@ -1,7 +1,10 @@ //! The step's `withdrawn` list: a step that cancels a task parked on a //! batch lists the calls it orphaned, the calls the same step issued -//! included, in effect-id order however many tasks it cancels, and a -//! decided run's step lists none. +//! included, in effect-id order however many tasks it cancels, and the +//! step that decides the run lists every effect still outstanding, of +//! every kind, the ones it issued included. + +use runvil_types::cancel::CancelHandle; use super::*; use crate::execute::tests::fixtures::echo_tool; @@ -38,17 +41,22 @@ const CANCELLED_CHILD: &str = "---\nname: loop\ndescription: d\nrunvil: 0\n---\n models.loop()\n\ return 'child done'\n```\n"; -/// A run of `source` over the loop models with `echo` offered, stepped by -/// hand. +/// A run of `source` over the loop models with `echo` offered, under the +/// cancel flag `cancel`, stepped by hand. #[track_caller] -fn echo_run(source: &str) -> Run { - loop_scenario(source).offer("echo", echo_tool()).into_run() +fn echo_run(source: &str, cancel: &CancelHandle) -> Run { + let cancel = cancel.clone(); + loop_scenario(source) + .offer("echo", echo_tool()) + .config(move |ctx| ctx.with_cancel(cancel)) + .into_run() } -/// A run of [`CANCELLED_CHILD`] past its first steps and Child's render, -/// with the ids of Main's `wait` round and Child's loop round. -fn cancelled_child() -> (Run, EffectId, EffectId) { - let mut run = echo_run(CANCELLED_CHILD); +/// A run of [`CANCELLED_CHILD`] under the cancel flag `cancel`, past its +/// first steps and Child's render, with the ids of Main's `wait` round and +/// Child's loop round. +fn cancelled_child(cancel: &CancelHandle) -> (Run, EffectId, EffectId) { + let mut run = echo_run(CANCELLED_CHILD, cancel); let (rounds, withdrawn) = step_past_renders(&mut run); assert!( withdrawn.is_empty(), @@ -86,7 +94,7 @@ fn tool_call_ids(effects: &[(EffectId, Effect)]) -> Vec { #[test] fn cancelling_a_task_withdraws_its_unanswered_tool_calls() { - let (mut run, infer, round) = cancelled_child(); + let (mut run, infer, round) = cancelled_child(&CancelHandle::new()); run.resume(round, two_echoes()); let (issued, withdrawn) = withdrawals(run.step()); let calls = tool_call_ids(&issued); @@ -114,7 +122,7 @@ fn cancelling_a_task_withdraws_its_unanswered_tool_calls() { } run.resume(linger, chat_answer(&text("done"))); let result = match run.step() { - Step::Done { result, .. } => result, + Step::Done { outcome, .. } => outcome, other @ Step::Pending { .. } => panic!("Main's second round ends the run: {other:?}"), }; assert_eq!(text_of(result), "cancelled"); @@ -122,7 +130,7 @@ fn cancelling_a_task_withdraws_its_unanswered_tool_calls() { #[test] fn a_call_issued_and_withdrawn_in_one_step_is_listed_twice() { - let (mut run, infer, round) = cancelled_child(); + let (mut run, infer, round) = cancelled_child(&CancelHandle::new()); run.resume(round, two_echoes()); run.resume(infer, chat_answer(&text("go"))); let (issued, withdrawn) = withdrawals(run.step()); @@ -141,41 +149,42 @@ fn a_call_issued_and_withdrawn_in_one_step_is_listed_twice() { run.resume(*id, answer); } let result = match run.step() { - Step::Done { result, .. } => result, + Step::Done { outcome, .. } => outcome, other @ Step::Pending { .. } => panic!("Main's second round ends the run: {other:?}"), }; assert_eq!(text_of(result), "cancelled"); } #[test] -fn a_decided_run_lists_no_withdrawn_calls() { - let (mut run, infer, round) = cancelled_child(); +fn the_step_that_decides_the_run_withdraws_every_outstanding_effect() { + let cancel = CancelHandle::new(); + let (mut run, infer, round) = cancelled_child(&cancel); run.resume(round, two_echoes()); let (issued, _) = withdrawals(run.step()); let calls = tool_call_ids(&issued); assert_eq!(calls.len(), 2, "both calls are in flight: {issued:?}"); - run.cancel(); + cancel.cancel(); let (issued, withdrawn) = withdrawals(run.step()); - assert!(run.decided(), "the cancel decides the run"); assert!( issued.is_empty(), "a torn-down run issues nothing: {issued:?}" ); - assert!( - withdrawn.is_empty(), - "the run's end abandoned Child, yet a decided run lists none: {withdrawn:?}" + assert_eq!( + withdrawn, + [infer, calls[0], calls[1]], + "the cancel withdraws Main's round and both of Child's calls, in effect-id order" ); - for id in calls.into_iter().chain([infer]) { + for id in withdrawn { run.resume(id, EffectAnswer::Dropped); } let result = match run.step() { - Step::Done { result, .. } => result, + Step::Done { outcome, .. } => outcome, other @ Step::Pending { .. } => { panic!("every orphan is answered, so the run is done: {other:?}") } }; assert!( - matches!(result, RunResult::Cancelled), + matches!(result, RunOutcome::Cancelled), "the run ends cancelled: {result:?}" ); } @@ -204,7 +213,7 @@ const CANCELLED_PAIR: &str = "---\nname: loop\ndescription: d\nrunvil: 0\n---\n\ #[test] fn one_step_lists_every_tasks_withdrawn_calls_in_effect_id_order() { - let mut run = echo_run(CANCELLED_PAIR); + let mut run = echo_run(CANCELLED_PAIR, &CancelHandle::new()); let (rounds, _) = step_past_renders(&mut run); let asking = |user: &str| { rounds @@ -245,8 +254,61 @@ fn one_step_lists_every_tasks_withdrawn_calls_in_effect_id_order() { }; run.resume(linger, chat_answer(&text("done"))); let result = match run.step() { - Step::Done { result, .. } => result, + Step::Done { outcome, .. } => outcome, other @ Step::Pending { .. } => panic!("Main's second round ends the run: {other:?}"), }; assert_eq!(text_of(result), "cancelled"); } + +/// Main spawns Store, Call, and Fast, waits on Fast under a timeout, then +/// cancels the other two and returns Fast's result, all in the run's first +/// step. +const DECIDED_IN_ONE_STEP: &str = "---\nname: loop\ndescription: d\nrunvil: 0\n---\n\n# Loop\n\n\ + ## Main\n\n```lua\n\ + local stored = tasks.spawn('## Store')\n\ + local called = tasks.spawn('## Call')\n\ + local fast = tasks.spawn('## Fast')\n\ + local _t, _ok, result = tasks.join_any({ fast }, { timeout = 30 })\n\ + tasks.cancel(stored)\n\ + tasks.cancel(called)\n\ + return result\n```\n\n\ + ## Store\n\n```lua\n\ + store.write('slow.md', 'never')\n\ + return 'stored'\n```\n\n\ + ## Call\n\n```lua\n\ + return tools.call('tools/echo', { value = 'never' })\n```\n\n\ + ## Fast\n\n```lua\n\ + return 'fast'\n```\n"; + +#[test] +fn the_step_that_decides_the_run_withdraws_the_effects_it_issued_of_every_kind() { + let mut run = echo_run(DECIDED_IN_ONE_STEP, &CancelHandle::new()); + let (issued, withdrawn) = withdrawals(run.step()); + let kinds: Vec<&str> = issued + .iter() + .map(|(_, effect)| match effect { + Effect::Timer { .. } => "timer", + Effect::Vfs { .. } => "store", + Effect::ToolCall { .. } => "tool call", + other => panic!("only the timer, the write, and the call: {other:?}"), + }) + .collect(); + assert_eq!( + kinds, + ["timer", "store", "tool call"], + "Main's timeout timer, then Store's write and Call's call" + ); + let ids: Vec = issued.iter().map(|(id, _)| *id).collect(); + assert_eq!( + withdrawn, ids, + "the step that decides the run withdraws every effect it issued, in effect-id order" + ); + for id in withdrawn { + run.resume(id, EffectAnswer::Dropped); + } + let result = match run.step() { + Step::Done { outcome, .. } => outcome, + other @ Step::Pending { .. } => panic!("every orphan is answered: {other:?}"), + }; + assert_eq!(text_of(result), "fast"); +} diff --git a/crates/runvil-internal/engine/src/execute/tests/parallel_tool_calls.rs b/crates/runvil-internal/engine/src/execute/tests/parallel_tool_calls.rs index 8890a048c..e33f4544e 100644 --- a/crates/runvil-internal/engine/src/execute/tests/parallel_tool_calls.rs +++ b/crates/runvil-internal/engine/src/execute/tests/parallel_tool_calls.rs @@ -4,7 +4,8 @@ //! records append in call order. An adhoc tool stops the batch and runs //! alone between its neighbors, and a task cancelled while parked on a //! batch orphans every call in it and lists each in the step's -//! `withdrawn`, which a decided run leaves empty. Each test steps a `Run` +//! `withdrawn`, where the step that decides the run lists every effect +//! still outstanding. Each test steps a `Run` //! by hand, so it sees exactly what each step issued and withdrew, and //! answers each section's render with its tools unchanged. @@ -152,7 +153,7 @@ fn a_rounds_catalog_calls_are_issued_in_one_step_and_append_in_call_order() { ); run.resume(next[0].0, chat_answer(&text("done"))); let result = match run.step() { - Step::Done { result, .. } => result, + Step::Done { outcome, .. } => outcome, other @ Step::Pending { .. } => panic!("the reply ends the section: {other:?}"), }; assert_eq!(text_of(result), "c1=first|c2=second"); @@ -224,7 +225,7 @@ fn adhoc_calls_run_alone_between_their_neighbors() { ); run.resume(next, chat_answer(&text("done"))); let result = match run.step() { - Step::Done { result, .. } => result, + Step::Done { outcome, .. } => outcome, other @ Step::Pending { .. } => panic!("the reply ends the section: {other:?}"), }; assert_eq!(text_of(result), "c1,c2,c3,c4|grabbed b"); @@ -280,7 +281,7 @@ fn cancelling_a_task_parked_on_a_batch_orphans_every_call() { assert!(issued.is_empty(), "both orphans are discarded: {issued:?}"); run.resume(linger, chat_answer(&text("done"))); let result = match run.step() { - Step::Done { result, .. } => result, + Step::Done { outcome, .. } => outcome, other @ Step::Pending { .. } => panic!("Main's second round ends the run: {other:?}"), }; assert_eq!(text_of(result), "cancelled"); diff --git a/crates/runvil-internal/engine/src/execute/tests/precheck_anchor.rs b/crates/runvil-internal/engine/src/execute/tests/precheck_anchor.rs index 6aed4936d..b73c0ad2d 100644 --- a/crates/runvil-internal/engine/src/execute/tests/precheck_anchor.rs +++ b/crates/runvil-internal/engine/src/execute/tests/precheck_anchor.rs @@ -178,43 +178,34 @@ fn a_merged_history_falls_back_to_the_estimate() { assert_eq!(gateway.call_count(), 1, "only the first round ran"); } -/// [`run_rounds`] with a third binding, `twin`: the default model under -/// other invocation settings, so it shares `writer`'s tokenizer. `other` +/// [`run_rounds`] with a third binding, `small`: the default model under +/// other invocation settings, so it shares `default`'s tokenizer. `strong` /// names a different model. #[track_caller] -fn run_rounds_with_twin(lua: &str, replies: Vec) -> (String, ScriptedChat) { - let mut models = loop_models(); - let writer = models.bindings()[0].clone(); - let twin = ModelBinding::new( - "twin", +fn run_rounds_with_small(lua: &str, replies: Vec) -> (String, ScriptedChat) { + let mut set = loop_models(); + let default = set.bindings()[0].clone(); + let small = ModelBinding::new( + "small", "The default model with other settings", - writer.id().clone(), + default.id().clone(), ModelInvocation::new().with_thinking(Some(false)), - writer.context(), + default.context(), ); - models.push(twin); - run_over(models, lua, replies) -} - -/// The model each request named, in the order they left. -fn request_models(gateway: &ScriptedChat) -> Vec { - gateway - .requests() - .iter() - .map(|body| body.options.model().to_owned()) - .collect() + set.push(small); + run_over(set, lua, replies) } #[test] fn a_model_switch_leaves_the_second_round_on_the_estimate() { - // The first round on `writer` anchors at 3500 tokens, which would + // The first round on `default` anchors at 3500 tokens, which would // refuse the second request (3604 plus the 512 reserve). The second - // request goes to `other`, a different model, so another tokenizer's + // request goes to `strong`, a different model, so another tokenizer's // count does not apply and the estimate of about 113 tokens admits it. let section = "messages:user('xxxxx')\n\ models.loop()\n\ messages:user(string.rep('y', 400))\n\ - models.use('other')\n\ + models.use('strong')\n\ local ok, err = pcall(models.loop)\n\ if ok then return 'sent' end\n\ return err.kind .. ':' .. err.reason"; @@ -224,38 +215,38 @@ fn a_model_switch_leaves_the_second_round_on_the_estimate() { ); assert_eq!(out, "sent"); assert_eq!(gateway.call_count(), 2, "both rounds left"); - assert_eq!(request_models(&gateway), ["test-model", "other-model"]); + assert_eq!(gateway.requested_models(), ["test-model", "other-model"]); } #[test] fn another_alias_of_the_same_model_keeps_the_anchor() { - // `twin` is a different alias with different settings, but it names the - // model that `writer` named, so the 3500-token anchor still refuses the + // `small` is a different alias with different settings, but it names the + // model that `default` named, so the 3500-token anchor still refuses the // second request. let section = "messages:user('xxxxx')\n\ models.loop()\n\ messages:user(string.rep('y', 400))\n\ - models.use('twin')\n\ + models.use('small')\n\ local ok, err = pcall(models.loop)\n\ if ok then return 'sent' end\n\ return err.kind .. ':' .. err.reason"; let (out, gateway) = - run_rounds_with_twin(section, vec![reply_with_usage("ok", 3400, 100, None)]); + run_rounds_with_small(section, vec![reply_with_usage("ok", 3400, 100, None)]); assert_eq!(out, "context_exhausted:precheck"); assert_eq!(gateway.call_count(), 1, "only the first round ran"); } #[test] fn the_next_served_round_anchors_on_the_new_model() { - // The switch to `other` runs on the estimate, because the first + // The switch to `strong` runs on the estimate, because the first // round's 3500 tokens would refuse it. That round reports 3500 tokens - // of its own. The third request stays on `other` and extends the + // of its own. The third request stays on `strong` and extends the // second round, so the new measurement anchors it and refuses it. The // estimate of the third request is about 220 tokens. let section = "messages:user('xxxxx')\n\ models.loop()\n\ messages:user(string.rep('y', 400))\n\ - models.use('other')\n\ + models.use('strong')\n\ models.loop()\n\ messages:user(string.rep('z', 400))\n\ local ok, err = pcall(models.loop)\n\ @@ -270,21 +261,21 @@ fn the_next_served_round_anchors_on_the_new_model() { ); assert_eq!(out, "context_exhausted:precheck"); assert_eq!(gateway.call_count(), 2, "only the first two rounds ran"); - assert_eq!(request_models(&gateway), ["test-model", "other-model"]); + assert_eq!(gateway.requested_models(), ["test-model", "other-model"]); } #[test] fn a_return_to_the_first_model_uses_the_estimate() { - // The second round on `other` replaced the measurement, so the third - // request back on `writer` meets another model's anchor and counts by + // The second round on `strong` replaced the measurement, so the third + // request back on `default` meets another model's anchor and counts by // the estimate. let section = "messages:user('xxxxx')\n\ models.loop()\n\ messages:user('yyyy')\n\ - models.use('other')\n\ + models.use('strong')\n\ models.loop()\n\ messages:user(string.rep('z', 400))\n\ - models.use('writer')\n\ + models.use('default')\n\ local ok, err = pcall(models.loop)\n\ if ok then return 'sent' end\n\ return err.kind .. ':' .. err.reason"; @@ -299,7 +290,7 @@ fn a_return_to_the_first_model_uses_the_estimate() { assert_eq!(out, "sent"); assert_eq!(gateway.call_count(), 3); assert_eq!( - request_models(&gateway), + gateway.requested_models(), ["test-model", "other-model", "test-model"] ); } @@ -314,9 +305,6 @@ fn run_with_max_tokens(lua: &str, max_tokens: u32) -> (String, ScriptedChat) { for binding in loop_set.bindings() { models.push(binding.clone().with_invocation(capped.clone())); } - if let Some(alias) = loop_set.default_alias() { - models.set_default_alias(alias.to_owned()); - } run_over(models, lua, vec![text("ok")]) } diff --git a/crates/runvil-internal/engine/src/execute/tests/preludes.rs b/crates/runvil-internal/engine/src/execute/tests/preludes.rs index b59572a88..25e8e32cb 100644 --- a/crates/runvil-internal/engine/src/execute/tests/preludes.rs +++ b/crates/runvil-internal/engine/src/execute/tests/preludes.rs @@ -1,5 +1,5 @@ //! Pack preludes in a run: every section VM installs the preludes -//! handed to `RunContext::with_preludes`, fanout arms and spawned task chains +//! handed to `RunConfig::with_preludes`, fanout arms and spawned task chains //! included, and the libraries a section requires see them; a prelude //! that fails to load or whose global collides fails the run as a Lua //! failure before it issues any effect; a store conflict a prelude's @@ -34,14 +34,14 @@ const EFFECT_FIRST: &str = "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n\ ## Only\n\n\ ```lua\nreturn 'done'\n```\n"; -/// Runs `md` prepared against the echo tool as its catalog, the test model, -/// and `preludes`. +/// Runs `md` over the echo tool as its catalog, the test model, and +/// `preludes`. #[track_caller] fn prelude_run(md: &str, preludes: Vec) -> Trace { Scenario::new(md) .bound() .tool(echo_tool()) - .context(move |ctx| ctx.with_preludes(preludes)) + .config(move |ctx| ctx.with_preludes(preludes)) .run() } @@ -61,9 +61,9 @@ fn records(trace: &Trace) -> Vec { fn assert_fails_before_any_effect(md: &str, preludes: Vec, fragments: &[&str]) { let control = prelude_run(md, Vec::new()); assert!( - matches!(control.result(), RunResult::Completed(_)), + matches!(control.outcome(), RunOutcome::Completed { .. }), "without preludes the prompt succeeds: {:?}", - control.result() + control.outcome() ); assert!( !control.effects().is_empty(), @@ -126,14 +126,14 @@ fn a_prelude_global_named_like_an_offered_tools_wire_name_installs() { #[test] fn a_prelude_global_named_like_a_model_role_label_installs() { - let md = "---\nname: t\ndescription: d\nrunvil: 0\nmodels:\n writer: {}\n---\n\n\ + let md = "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n\ # Preludes\n\n\ ## Only\n\n\ - ```lua\nreturn writer .. '|' .. models.get('writer').name\n```\n"; - let trace = prelude_run(md, vec![prelude("writer", "writer = 'mine'")]); + ```lua\nreturn strong .. '|' .. models.get('strong').name\n```\n"; + let trace = prelude_run(md, vec![prelude("strong", "strong = 'mine'")]); assert_eq!( trace.text(), - "mine|writer", + "mine|strong", "the prelude's global and the role's handle sit side by side" ); } @@ -198,16 +198,16 @@ fn a_prelude_write_across_two_arms_ends_the_run_with_determinism_caught_or_not() "pcall(function() store.write('seeded.txt', 'seed') end)", ] { let trace = prelude_run(md, vec![prelude("seeder", source)]); - let error = match trace.result() { - RunResult::Failure(error) => error, + let (kind, message) = match trace.outcome() { + RunOutcome::Failed { kind, message } => (*kind, message), other => { panic!("the second arm's install-time write ends the run: `{source}`: {other:?}") } }; - assert_eq!( - error.kind(), - RunErrorKind::Determinism, - "the conflict wins over the prelude's own outcome: `{source}`: {error:?}" + assert_eq!(kind, RunErrorKind::Vfs, "`{source}`: {message}"); + assert!( + message.starts_with("store determinism violation: "), + "the conflict wins over the prelude's own outcome: `{source}`: {message}" ); } } diff --git a/crates/runvil-internal/engine/src/execute/tests/render_effect.rs b/crates/runvil-internal/engine/src/execute/tests/render_effect.rs index 179858c77..96b8e22d0 100644 --- a/crates/runvil-internal/engine/src/execute/tests/render_effect.rs +++ b/crates/runvil-internal/engine/src/execute/tests/render_effect.rs @@ -80,7 +80,7 @@ fn a_section_with_catalog_offers_renders_once_before_its_first_chat_and_every_ro let EffectRecord::Render { alias, offered, .. } = &rig.issued()[0] else { panic!("the first effect is the render: {:?}", rig.issued()); }; - assert_eq!(alias, "writer", "the render names the round's slot"); + assert_eq!(alias, "default", "the render names the round's slot"); assert_eq!( *offered, vec![PackToolSets::new( @@ -218,7 +218,7 @@ fn a_handle_round_renders_and_then_chats_on_the_handles_binding() { let source = one_section( "tools.offer('tools/echo') messages:user('go') - models.get('other'):loop() + models.loop(models.get('strong')) return messages[#messages].content", ); let rig = Rig::run( @@ -238,12 +238,12 @@ fn a_handle_round_renders_and_then_chats_on_the_handles_binding() { .collect(); assert_eq!( slots, - [("render", "other"), ("chat", "other")], + [("render", "strong"), ("chat", "strong")], "the round keeps the handle's binding, never the section's model" ); assert_eq!( gateway.requests()[0].options.model(), - loop_model_name("other") + loop_model_name("strong") ); } @@ -253,11 +253,11 @@ fn a_raw_gateway_model_id_renders_with_a_model_info_built_from_its_binding() { let source = one_section( "tools.offer('tools/echo') messages:user('go') - models.get('raw-model'):loop() + models.loop(models.get('raw-model')) return messages[#messages].content", ); let mut driving = loop_scenario(source) - .context(|ctx| ctx.with_ui(json!({}))) + .config(|ctx| ctx.with_ui(json!({}))) .offer("echo", echo_tool()) .chat(&gateway) .start(); @@ -298,7 +298,7 @@ fn rendered_run( } other => panic!("only the render and the round are issued: {other:?}"), }; - log.push((effect.record(), answer.record())); + log.push((effect.to_record(), answer.to_record())); answer }); assert_eq!(text_of(result), "done"); @@ -369,7 +369,7 @@ fn a_dropped_render_answers_its_round_as_cancelled_and_fixes_no_tools() { let run = loop_scenario(source).offer("echo", echo_tool()).into_run(); let mut issued = Vec::new(); let (result, _) = drive(run, |_, effect| { - issued.push(effect.record()); + issued.push(effect.to_record()); match effect { Effect::Render { .. } if issued.len() == 1 => EffectAnswer::Dropped, Effect::Render { offered, .. } => EffectAnswer::Render(offered.clone()), diff --git a/crates/runvil-internal/engine/src/execute/tests/rendered_tools.rs b/crates/runvil-internal/engine/src/execute/tests/rendered_tools.rs index 4c637535c..37623a23a 100644 --- a/crates/runvil-internal/engine/src/execute/tests/rendered_tools.rs +++ b/crates/runvil-internal/engine/src/execute/tests/rendered_tools.rs @@ -8,7 +8,7 @@ use runvil_parser::test_support::{one_section, prompt}; use super::fixtures::{echo_tool, scoped_tool}; -use super::models_loop::{catalog_scenario, loop_models, loop_scenario}; +use super::models_loop::{loop_models, loop_scenario}; use super::*; use crate::model::{ModelBinding, ModelId, ModelInvocation, ToolSchema}; use crate::test_support::scenario::test_binding; @@ -17,12 +17,12 @@ use crate::test_support::scenario::test_binding; const FIXED: &str = "the section's tool list is fixed at its first models.loop round; make \ every offer before the first loop, or offer in another section"; -/// The loop models plus `wide`, a model whose window is large enough for -/// a request the 4096-token `writer` refuses. +/// The loop models plus `creative`, a model whose window is large enough for +/// a request the 4096-token `default` refuses. fn wide_models() -> ModelSet { let mut models = loop_models(); models.push(ModelBinding::new( - "wide", + "creative", "A model with a wide window", ModelId::gateway("wide-model").expect("the test model id is valid"), ModelInvocation::new(), @@ -51,7 +51,7 @@ fn a_first_round_the_precheck_refuses_still_fixes_the_tool_list() { return '' end)) assert(not offered, 'the refused round fixed the list') - models.use('wide') + models.use('creative') models.loop() return refusal.kind .. ':' .. tostring(refusal)", ); @@ -86,7 +86,7 @@ fn every_later_round_sends_the_same_tool_bytes_across_a_model_change() { messages:user('one') models.loop() messages:user('two') - models.get('other'):loop() + models.loop(models.get('strong')) return messages[#messages].content", ); let trace = loop_scenario(source) @@ -141,10 +141,7 @@ fn after_the_render_every_offer_raises_even_an_identical_one_while_defining_stil ("offers", "tools.offer('tools/echo')"), ], ); - let trace = catalog_scenario(source) - .tool(echo_tool()) - .chat(&gateway) - .run(); + let trace = loop_scenario(source).tool(echo_tool()).chat(&gateway).run(); let lines: Vec<&str> = trace.text().lines().collect(); let offer = format!("lua:tools.offer: {FIXED}"); assert_eq!( @@ -238,12 +235,12 @@ fn a_first_round_refused_for_a_missing_model_renders_nothing() { local ok, err = pcall(models.loop) assert(not ok, 'a section with no model is refused') tools.offer('tools/echo') - models.get('writer'):loop() + models.loop(models.get('strong')) return tostring(err)", ); let mut unset = ModelSet::default(); unset.push(test_binding( - "writer", + "strong", "A general model for tests", "test-model", )); diff --git a/crates/runvil-internal/engine/src/execute/tests/run_inputs.rs b/crates/runvil-internal/engine/src/execute/tests/run_inputs.rs index e336d4cfe..bb06a8549 100644 --- a/crates/runvil-internal/engine/src/execute/tests/run_inputs.rs +++ b/crates/runvil-internal/engine/src/execute/tests/run_inputs.rs @@ -1,14 +1,13 @@ -//! The caller-drawn inputs on `RunContext`: `seed` (the untrusted-envelope +//! The caller-drawn inputs on `RunConfig`: `seed` (the untrusted-envelope //! nonce derives from it), `started_at` (rendered as `sys.when` for the H1 //! pass and every walked section alike), and `ui` (the snapshot the `ui()` //! global serves). Two runs under the same inputs agree byte for byte, and //! `sys.now` is an unknown field. -use runvil_types::replay::Flags; -use runvil_types::seed::{RunSeed, RunUid}; +use runvil_types::seed::RunSeed; use runvil_types::timestamp::Timestamp; -use super::context::{parse, test_context}; +use super::context::{parse, test_config}; use super::*; use crate::execute::run::Run; use crate::test_support::drive; @@ -26,12 +25,12 @@ const DRAWS: u64 = 0x5EED; /// Runs `md` with no effects under `seed` and [`STARTED_AT`], returning /// the run's text. -fn run_seeded(md: &str, seed: RunSeed) -> RunResult { - let ctx = RunContext::new(EXECUTION, seed, STARTED_AT); - let (result, _) = drive(Run::new(Arc::new(parse(md)), "", ctx), |_, effect| { +fn run_seeded(md: &str, seed: RunSeed) -> RunOutcome { + let ctx = RunConfig::new(EXECUTION, seed, STARTED_AT); + let (outcome, _) = drive(Run::new(Arc::new(parse(md)), "", ctx), |_, effect| { panic!("no effect is issued: {effect:?}") }); - result + outcome } /// The nonce between `` in a wrapped envelope. @@ -63,7 +62,7 @@ fn two_runs_with_the_same_seed_and_started_at_produce_identical_nonces_and_sys_w ); assert!( first.starts_with(&format!("{STARTED_AT_RFC3339}|")), - "sys.when is the context's started_at rendered as RFC 3339: {first}" + "sys.when is the config's started_at rendered as RFC 3339: {first}" ); assert_eq!( nonce_in(&first).len(), @@ -115,7 +114,7 @@ fn sys_when_is_timestamp_to_rfc3339_for_any_started_at() { return sys.when\n\ ```\n"; let whole_second = Timestamp::from_unix_millis(1_700_000_000_000); - let ctx = RunContext::new(EXECUTION, TEST_SEED, whole_second); + let ctx = RunConfig::new(EXECUTION, TEST_SEED, whole_second); let (result, _) = drive(Run::new(Arc::new(parse(md)), "", ctx), |_, effect| { panic!("no effect is issued: {effect:?}") }); @@ -128,35 +127,11 @@ fn sys_when_is_timestamp_to_rfc3339_for_any_started_at() { } #[test] -fn the_context_holds_the_flags_and_starts_them_empty() { - // `Flags` is a run input like the seed: empty from `new`, kept - // verbatim when the caller sets it, and readable beside the other - // inputs. - let fresh = test_context(EXECUTION); - assert_eq!(fresh.flags(), Flags::EMPTY); - assert!(fresh.flags().is_empty(), "no flag is set by default"); - - let recorded = Flags::from_bits(0b101); +fn a_run_configs_debug_shows_no_seed() { let mut rng = Seeded::new(DRAWS); let seed = rng.run_seed(); - let ctx = RunContext::new(EXECUTION, seed, STARTED_AT).with_flags(recorded); - assert_eq!(ctx.flags(), recorded, "the caller's flags are kept"); - rng.check_eq(&ctx.seed(), &seed, "the caller's seed is kept"); - assert_eq!(ctx.started_at(), STARTED_AT); -} - -#[test] -fn a_run_contexts_debug_shows_the_run_id_and_not_the_seed() { - let mut rng = Seeded::new(DRAWS); - let seed = rng.run_seed(); - let ctx = RunContext::new(EXECUTION, seed, STARTED_AT); + let ctx = RunConfig::new(EXECUTION, seed, STARTED_AT); let debug = format!("{ctx:?}"); - let uid = RunUid::from_seed(&seed); - rng.check_eq(&ctx.uid(), &uid, "the run id derives from the seed"); - rng.check( - debug.contains(&format!("uid: {uid:?}")), - format_args!("the Debug names the run id {uid}: {debug}"), - ); rng.check( !debug.contains(&seed.to_string()) && !debug.to_lowercase().contains("seed"), format_args!("the Debug shows no seed: {debug}"), @@ -171,13 +146,13 @@ fn sys_now_is_absent_from_the_globals() { ```lua\n\ return sys.now\n\ ```\n"; - let error = match run_seeded(md, TEST_SEED) { - RunResult::Failure(error) => error, + let message = match run_seeded(md, TEST_SEED) { + RunOutcome::Failed { message, .. } => message, other => panic!("reading sys.now fails the section: {other:?}"), }; assert!( - error.to_string().contains("unknown sys field 'now'"), - "sys.now is an unknown field, not a stale clock: {error}" + message.contains("unknown sys field 'now'"), + "sys.now is an unknown field, not a stale clock: {message}" ); } @@ -190,7 +165,7 @@ fn the_ui_global_serves_the_snapshot_taken_at_run_start() { return ui().selected_model .. '/' .. tostring(ui().workspace_root)\n\ ```\n"; let ctx = - test_context(EXECUTION).with_ui(json!({ "selected_model": "m-1", "workspace_root": null })); + test_config(EXECUTION).with_ui(json!({ "selected_model": "m-1", "workspace_root": null })); let (result, _) = drive(Run::new(Arc::new(parse(md)), "", ctx), |_, effect| { panic!("no effect is issued: {effect:?}") }); diff --git a/crates/runvil-internal/engine/src/execute/tests/run_termination.rs b/crates/runvil-internal/engine/src/execute/tests/run_termination.rs index c076b8546..0afca02a8 100644 --- a/crates/runvil-internal/engine/src/execute/tests/run_termination.rs +++ b/crates/runvil-internal/engine/src/execute/tests/run_termination.rs @@ -44,9 +44,9 @@ fn cancelling_a_run_settles_every_live_task_with_one_terminal_before_the_run_end let trace = driving.finish(); assert!( - matches!(trace.result(), RunResult::Cancelled), + matches!(trace.outcome(), RunOutcome::Cancelled), "the test driver's cancel ends the run as cancelled: {:?}", - trace.result() + trace.outcome() ); assert!( matches!( diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler.rs index 814c65f7c..9fbfadad5 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler.rs @@ -1,7 +1,9 @@ //! Scheduler-side tests, split by seam into the `walk`, `h1`, `fanout`, -//! `concurrency`, and `failures` submodules. The model set `writer_models`, -//! its scenario `scheduler_scenario`, the concurrency ceiling `ceiling`, -//! the round and timer matchers `asks` and `is_timer`, the prompt readers +//! `concurrency`, `chain_model`, `fanout_model`, and `failures` submodules. +//! The model sets `default_models` and `three_models`, their scenarios +//! `scheduler_scenario` and `chain_scenario`, the section builder `sections`, +//! the concurrency ceiling `ceiling`, the round and timer matchers `asks` and +//! `is_timer`, the prompt readers //! `request_prompts` and `issued_prompts`, the replaying `recorded`, and //! the task-lifecycle counter `terminal_count` with its event matchers live //! here because more than one submodule uses them. @@ -18,24 +20,50 @@ use crate::execute::EffectRecord; use crate::test_support::rig::RigTrace; use crate::test_support::scenario::{infer_prompt, test_binding}; -/// The model set the H1 pass would leave behind: one `writer` binding -/// as the prompt-wide default. The scheduler's tests bypass H1, so they -/// pre-fill the run's shared set directly. -fn writer_models() -> ModelSet { +/// The model set a run with a one-model role map holds: one `default` +/// binding, which every section starts on. The scheduler's tests give no +/// role map, so they pre-fill the run's shared set directly. +fn default_models() -> ModelSet { let mut set = ModelSet::default(); set.push(test_binding( - "writer", + "default", "A general model for tests", "test-model", )); - set.set_default_alias("writer".to_owned()); set } -/// A run of `source` over the [`writer_models`], unprepared, so the run -/// starts with that set and no catalog. +/// A run of `source` over the [`default_models`], so the run starts with +/// that set. pub(super) fn scheduler_scenario(source: impl Into) -> Scenario { - Scenario::new(source).unprepared().models(writer_models()) + Scenario::new(source).models(default_models()) +} + +/// The model set the chain and fanout tests run over: the `default` binding +/// on `test-model`, `strong` on `model-b`, and `fast` on `model-c`. +pub(super) fn three_models() -> ModelSet { + let mut set = default_models(); + set.push(test_binding("strong", "A second model", "model-b")); + set.push(test_binding("fast", "A third model", "model-c")); + set +} + +/// A run of `md` over the [`three_models`], so the run starts with that +/// set. +pub(super) fn chain_scenario(md: &str) -> Scenario { + Scenario::new(md).models(three_models()) +} + +/// A prompt with the given sections, each `(heading, lua)`, under one title. +pub(super) fn sections(sections: &[(&str, &str)]) -> String { + let mut md = String::from("---\nname: t\ndescription: d\nrunvil: 0\n---\n\n# Chain\n\n"); + for (heading, lua) in sections { + md.push_str(heading); + md.push_str("\n\n```lua\n"); + md.push_str(lua); + md.push_str("\n```\n\n"); + } + md } /// The run's concurrency ceiling narrowed to `n` admitted tasks. @@ -126,8 +154,10 @@ fn terminal_count(trace: &Trace, is: fn(&Event) -> bool) -> usize { trace.events().iter().filter(|event| is(event)).count() } +mod chain_model; mod concurrency; mod failures; mod fanout; +mod fanout_model; mod h1; mod walk; diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/chain_model.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/chain_model.rs new file mode 100644 index 000000000..f4342b4d3 --- /dev/null +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/chain_model.rs @@ -0,0 +1,158 @@ +//! A chain's current model: `models.use` lasts across the chain's sections, +//! a `call` child and a spawned task start with a copy of the parent's model +//! taken when they are dispatched, a child's switch never reaches its parent +//! or its siblings, and the root walk starts on the model the H1 pass ended +//! on. + +use super::*; +use crate::execute::tests::fixtures::echo_tool; + +#[test] +fn a_switch_in_one_section_is_still_in_effect_in_the_next() { + let gateway = ScriptedChat::new(vec![text("r")]); + let md = sections(&[ + ("## One", "models.use('strong')"), + ( + "## Two", + "local label = models.current().label\nreturn label .. '|' .. models.infer('go')", + ), + ]); + let trace = chain_scenario(&md).chat(&gateway).run(); + assert_eq!(trace.text(), "strong|r"); + assert_eq!(gateway.requested_models(), ["model-b"]); +} + +#[test] +fn a_prompt_that_never_switches_reads_default_in_every_section() { + let md = sections(&[ + ("## One", "var.first = models.current().label"), + ( + "## Two", + "return var.first .. '|' .. models.current().label", + ), + ]); + let trace = chain_scenario(&md).run(); + assert_eq!(trace.text(), "default|default"); +} + +#[test] +fn sys_model_at_the_first_tool_call_of_a_later_section_reflects_the_earlier_switch() { + let md = sections(&[ + ("## One", "models.use('strong')"), + ( + "## Two", + "tools.call('tools/echo', { value = 'y' })\nreturn sys.model", + ), + ]); + let trace = chain_scenario(&md).offer("echo", echo_tool()).run(); + assert_eq!(trace.text(), "model-b"); +} + +#[test] +fn a_call_child_and_a_spawned_task_inherit_the_parents_switch() { + let md = sections(&[ + ( + "## Main", + "models.use('strong')\n\ + local viaCall = call('## Child')\n\ + local t = tasks.spawn('## Child')\n\ + local _, ok, viaTask = tasks.join_any({ t })\n\ + assert(ok)\n\ + return viaCall .. '|' .. viaTask", + ), + ("## Child", "return models.current().label"), + ]); + let trace = chain_scenario(&md).run(); + assert_eq!(trace.text(), "strong|strong"); +} + +#[test] +fn a_childs_switch_never_reaches_its_parent_or_a_sibling() { + let md = sections(&[ + ( + "## Main", + "models.use('strong')\n\ + local viaCall = call('## Switcher')\n\ + local after = models.current().label\n\ + local a = tasks.spawn('## Switcher')\n\ + local b = tasks.spawn('## Reader')\n\ + local _, okA, labelA = tasks.join_any({ a })\n\ + local _, okB, labelB = tasks.join_any({ b })\n\ + assert(okA and okB)\n\ + return viaCall .. '|' .. after .. '|' .. labelA .. '|' .. labelB .. '|' .. models.current().label", + ), + ( + "## Switcher", + "models.use('fast')\nreturn models.current().label", + ), + ("## Reader", "return models.current().label"), + ]); + let trace = chain_scenario(&md).run(); + assert_eq!(trace.text(), "fast|strong|fast|strong|strong"); +} + +#[test] +fn a_switch_in_h1_is_the_starting_model_of_the_root_walk() { + let md = sections(&[ + ("", "models.use('strong')"), + ("## Main", "return models.current().label"), + ]); + let trace = chain_scenario(&md).run(); + assert_eq!(trace.text(), "strong"); +} + +#[test] +fn a_switch_in_h1_reaches_the_walk_a_jump_starts() { + let md = sections(&[ + ("", "models.use('strong')\njump('## Target')"), + ("## Skipped", "error('the jump must skip this section')"), + ("## Target", "return models.current().label"), + ]); + let trace = chain_scenario(&md).run(); + assert_eq!(trace.text(), "strong"); +} + +#[test] +fn a_queued_task_keeps_the_model_its_parent_had_when_it_spawned_it() { + // `tasks.concurrency(1)` admits the first child, which parks on its model + // round, so the second stays queued while the parent switches to `fast`. + // The second child reads the model the parent had at the spawn. + let gateway = ScriptedChat::new(vec![text("r1"), text("r2")]); + let md = sections(&[ + ( + "## Main", + "tasks.concurrency(1)\n\ + models.use('strong')\n\ + local a = tasks.spawn('## Child')\n\ + local b = tasks.spawn('## Child')\n\ + models.use('fast')\n\ + local _, okA, first = tasks.join_any({ a })\n\ + local _, okB, second = tasks.join_any({ b })\n\ + assert(okA and okB)\n\ + return first .. '|' .. second", + ), + ( + "## Child", + "local label = models.current().label\nreturn label .. ':' .. models.infer('x')", + ), + ]); + let trace = chain_scenario(&md).limits(ceiling(4)).chat(&gateway).run(); + assert_eq!(trace.text(), "strong:r1|strong:r2"); + assert_eq!(gateway.requested_models(), ["model-b", "model-b"]); +} + +#[test] +fn all_arms_of_one_fanout_start_on_the_same_model() { + let md = sections(&[ + ( + "## Main", + "tasks.concurrency(1)\n\ + models.use('strong')\n\ + local r = tasks.fanout('### Arm', {'a', 'b', 'c'})\n\ + return table.concat(r, ',')", + ), + ("### Arm", "return models.current().label"), + ]); + let trace = chain_scenario(&md).limits(ceiling(4)).run(); + assert_eq!(trace.text(), "strong,strong,strong"); +} diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/concurrency-stepped.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/concurrency-stepped.rs index 87a5942a3..4e47b5cd8 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler/concurrency-stepped.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/concurrency-stepped.rs @@ -3,7 +3,7 @@ //! queue that can never be admitted reported as a stall. use super::*; -use crate::execute::RunResult; +use crate::execute::RunOutcome; use crate::test_support::rig::Rig; use crate::test_support::scenario::{Batching, drive_batched}; @@ -36,7 +36,9 @@ fn two_tasks_whose_call_children_finish_in_any_order_both_complete() { let run = scheduler_scenario(md).limits(ceiling(2)).into_run(); let outcome = drive_batched(run, batching); match &outcome.result { - RunResult::Completed(text) => assert_eq!(text, "r(a)|r(b)", "under {batching:?}"), + RunOutcome::Completed { final_text } => { + assert_eq!(final_text, "r(a)|r(b)", "under {batching:?}"); + } other => panic!("both calls complete under {batching:?}: {other:?}"), } } @@ -73,11 +75,11 @@ fn a_queued_task_that_can_never_be_admitted_is_reported_as_a_stall() { "a stalled run must not return Pending with more effects: {:?}", rig.issued() ); - match rig.trace().result() { - RunResult::Failure(error) => assert!( - error.to_string().contains("stalled"), - "the run reports the stall: {error}" + match rig.trace().outcome() { + RunOutcome::Failed { message, .. } => assert!( + message.contains("stalled"), + "the run reports the stall: {message}" ), - result => panic!("the stalled run must fail, got {result:?}"), + outcome => panic!("the stalled run must fail, got {outcome:?}"), } } diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/failures-script-tools.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/failures-script-tools.rs index 43dd14f1b..abbab0930 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler/failures-script-tools.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/failures-script-tools.rs @@ -4,7 +4,9 @@ //! wrapping, and the model install around a call. use super::*; -use crate::execute::tests::fixtures::{echo_tool, structured_tool, untrusted_echo_tool}; +use crate::execute::tests::fixtures::{ + echo_tool, model_only_tool, structured_tool, untrusted_echo_tool, +}; #[test] fn a_script_tools_call_dispatches_and_resumes_as_a_string() { @@ -52,7 +54,7 @@ fn a_script_tools_call_with_an_unbound_name_lists_the_offered_ids() { ## Only\n\n\ ```lua\ntools.offer('tools/echo')\nreturn tools.call('tools/missing', {})\n```\n"; let trace = scheduler_scenario(md).offer("echo", echo_tool()).run(); - match trace.error().inner() { + match trace.error() { Error::UnboundToolCall { name, ids } => { assert_eq!(name, "tools/missing"); assert_eq!(ids, &["tools/echo".to_owned()]); @@ -78,6 +80,90 @@ fn a_script_tools_call_reaches_an_offered_tool_outside_the_section_scope() { assert_eq!(trace.text(), "echoed: hi|1"); } +#[test] +fn a_script_tools_call_of_a_model_only_tool_is_refused_naming_it() { + // The Pack marked the tool model-only, so the catalog still names it + // and a section may offer it, while a script's call fails at + // dispatch: only a model receives arguments that suit the schema a + // render answered. + let md = "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n\ + # ToolCall\n\n\ + ## Only\n\n\ + ```lua\n\ + tools.offer('tools/echo')\n\ + return tools.call('tools/replace', { value = 'hi' })\n\ + ```\n"; + let trace = scheduler_scenario(md) + .offer("echo", echo_tool()) + .offer("replace", model_only_tool()) + .run(); + match trace.error() { + Error::ModelOnlyToolCall { name, id } => { + assert_eq!(name, "tools/replace"); + assert_eq!(id, "tools/replace"); + } + other => panic!("expected the typed model-only error, got {other:?}"), + } +} + +#[test] +fn a_script_pcall_catches_the_model_only_refusal_and_the_run_goes_on() { + // The refusal is a Lua error like any other failed `tools.call`, so + // an author can catch it and offer the tool instead. + let md = "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n\ + # ToolCall\n\n\ + ## Only\n\n\ + ```lua\n\ + tools.offer('tools/replace')\n\ + local ok = pcall(tools.call, 'tools/replace', { value = 'hi' })\n\ + assert(not ok, 'the model-only call is refused')\n\ + return 'caught'\n\ + ```\n"; + let trace = scheduler_scenario(md) + .offer("replace", model_only_tool()) + .run(); + assert_eq!(trace.text(), "caught", "the run continues after the catch"); +} + +#[test] +fn a_script_tools_call_of_a_model_only_tool_by_its_object_is_refused_the_same_way() { + // The object form resolves through the same lookup as the id, so the + // refusal cannot be skipped by holding the tool object. + let md = "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n\ + # ToolCall\n\n\ + ## Only\n\n\ + ```lua\n\ + tools.offer('tools/replace')\n\ + local tool = tools.get('tools/replace')\n\ + return tools.call(tool, { value = 'hi' })\n\ + ```\n"; + let trace = scheduler_scenario(md) + .offer("replace", model_only_tool()) + .run(); + match trace.error() { + Error::ModelOnlyToolCall { id, .. } => assert_eq!(id, "tools/replace"), + other => panic!("expected the typed model-only error, got {other:?}"), + } +} + +#[test] +fn a_model_only_tool_is_still_offerable_alone_and_with_its_pack() { + // The flag forbids the call, never the offer: a prompt offers the + // tool and the model calls it. + let md = "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n\ + # ToolCall\n\n\ + ## Only\n\n\ + ```lua\n\ + tools.offer('tools/replace')\n\ + tools.offer('tools')\n\ + return 'offered'\n\ + ```\n"; + let trace = scheduler_scenario(md) + .offer("replace", model_only_tool()) + .run(); + assert_eq!(trace.text(), "offered"); +} + #[test] fn cancellation_interrupts_a_slow_script_tools_call() { let md = "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n\ @@ -99,9 +185,9 @@ fn cancellation_interrupts_a_slow_script_tools_call() { .run(); assert!( - matches!(trace.result(), RunResult::Cancelled), + matches!(trace.outcome(), RunOutcome::Cancelled), "cancelling a suspended tools.call must interrupt the run, got {:?}", - trace.result() + trace.outcome() ); assert_eq!( started.load(Ordering::SeqCst), @@ -161,7 +247,7 @@ fn invalid_json_from_a_structured_tool_is_a_tool_error() { let trace = scheduler_scenario(md) .offer("form", structured_tool("not json", true)) .run(); - match trace.error().inner() { + match trace.error() { Error::Tool { message, .. } => { assert!( message.contains("returned invalid JSON"), @@ -185,7 +271,7 @@ fn an_untrusted_structured_output_is_wrapped_before_classification() { let trace = scheduler_scenario(md) .offer("form", structured_tool("{\"text\":\"typed\"}", false)) .run(); - match trace.error().inner() { + match trace.error() { Error::Tool { message, .. } => { assert!( message.contains("returned invalid JSON"), @@ -216,36 +302,30 @@ fn a_script_tools_call_before_infer_keeps_the_model_install() { } #[test] -fn a_first_tool_call_whose_model_step_fails_leaves_the_install_to_the_next_call() { - // The set's default names no binding, so the first call's model step - // fails after its counts install. The install is not done until the - // model step succeeds, so the second call runs it again, under the - // selection `models.use` made in between, and `sys.model` lands. +fn a_first_tool_call_sets_sys_model_to_the_model_models_use_selected() { + // The first tool call takes the section's current model, so the switch + // `models.use` made before it is what `sys.model` names. let md = "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n\ # ToolCall\n\n\ ## Only\n\n\ ```lua\n\ - local ok = pcall(tools.call, 'tools/echo', { value = 'x' })\n\ - assert(not ok, 'the first call fails in its model step')\n\ - models.use('writer')\n\ + models.use('strong')\n\ tools.call('tools/echo', { value = 'y' })\n\ return sys.model\n\ ```\n"; - let mut unbound_default = ModelSet::default(); - unbound_default.push(test_binding( - "writer", + let mut models = ModelSet::default(); + models.push(test_binding( + "default", "A general model for tests", "test-model", )); - unbound_default.set_default_alias("ghost".to_owned()); + models.push(test_binding("strong", "A second model", "other-model")); let trace = Scenario::new(md) - .unprepared() - .models(unbound_default) + .models(models) .offer("echo", echo_tool()) .run(); - assert_eq!(trace.text(), "test-model"); + assert_eq!(trace.text(), "other-model"); } - #[test] fn a_document_prompt_without_tools_call_is_unaffected() { // Bindings installed, shim present, `tools.call` never called: the diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/failures.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/failures.rs index b2c6edf2d..c66446b2e 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler/failures.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/failures.rs @@ -28,7 +28,7 @@ fn two_arms_appending_one_path_boom_without_any_other_suspension() { ```\n"; let trace = scheduler_scenario(md).run(); - match trace.error().inner() { + match trace.error() { Error::Determinism(detail) => { assert!(detail.contains("log.txt"), "error was: {detail}"); } @@ -108,7 +108,7 @@ fn fatal_arm_aborts_queued_siblings() { ```\n"; let trace = scheduler_scenario(md).limits(ceiling(1)).run(); - let error = trace.error().inner(); + let error = trace.error(); assert!( !matches!(error, Error::Interrupted), "expected a fatal arm error, got {error}" @@ -158,7 +158,7 @@ fn fatal_arm_aborts_an_in_flight_sibling() { return a\n\ ```\n"; let rig = Rig::run(scheduler_scenario(md).chat(&gateway).hold(asks("slow:1"))); - let error = rig.trace().error().inner(); + let error = rig.trace().error(); assert!( !matches!(error, Error::Interrupted), @@ -265,9 +265,9 @@ fn cancellation_while_suspended_in_a_fanout_arm_interrupts_the_run() { let rig = rig.finish(); assert!( - matches!(rig.trace().result(), RunResult::Cancelled), + matches!(rig.trace().outcome(), RunOutcome::Cancelled), "cancelling suspended arms must interrupt the run, got {:?}", - rig.trace().result() + rig.trace().outcome() ); assert_eq!( issued_prompts(&rig), @@ -391,7 +391,7 @@ fn an_answer_for_an_unknown_request_id_fails_loudly() { let rig = rig.finish(); assert!( - matches!(rig.trace().error().inner(), Error::Internal { message, .. } if message.contains("did not issue")), + matches!(rig.trace().error(), Error::Internal { message, .. } if message.contains("did not issue")), "the unknown answer is a loud invariant failure: {:?}", rig.trace().error() ); @@ -418,7 +418,7 @@ fn aborting_an_effect_that_is_not_outstanding_records_no_orphan() { let rig = rig.finish(); assert!( - matches!(rig.trace().error().inner(), Error::Internal { message, .. } if message.contains("did not issue")), + matches!(rig.trace().error(), Error::Internal { message, .. } if message.contains("did not issue")), "the answer for the never-issued id still fails loudly: {:?}", rig.trace().error() ); diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/fanout/arms.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/fanout/arms.rs index 029eba697..336661f02 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler/fanout/arms.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/fanout/arms.rs @@ -22,9 +22,9 @@ fn pre_cancelled_fanout_returns_interrupted() { driving.cancel(); let trace = driving.finish(); assert!( - matches!(trace.result(), RunResult::Cancelled), + matches!(trace.outcome(), RunOutcome::Cancelled), "a pre-cancelled fanout must interrupt the run, got {:?}", - trace.result() + trace.outcome() ); } @@ -32,7 +32,7 @@ fn pre_cancelled_fanout_returns_interrupted() { fn model_required_when_arm_infer_has_no_binding() { // An arm whose explicit infer of its prose has no model binding fails // the fanout with Error::ModelRequired naming the worker section. The - // run starts unprepared with nothing filled in, so the model set stays + // run has no role map and nothing filled in, so the model set stays // empty - the scheduler scenario pre-fills a default binding. let md = "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n\ # Fanout\n\n\ @@ -43,10 +43,10 @@ fn model_required_when_arm_infer_has_no_binding() { ### Worker\n\n\ Ask the model about {{ item }}.\n\n\ ```lua\nreturn models.infer(prose)\n```\n"; - let trace = Scenario::new(md).unprepared().run(); + let trace = Scenario::new(md).run(); let error = trace.error(); assert!( - matches!(error.inner(), Error::ModelRequired { .. }), + matches!(error, Error::ModelRequired { .. }), "expected ModelRequired, got {error}" ); assert!( diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/fanout/failure_semantics.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/fanout/failure_semantics.rs index 7882a6c3d..48ff05f50 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler/fanout/failure_semantics.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/fanout/failure_semantics.rs @@ -91,7 +91,7 @@ fn fanout_depth_cap_reads_the_chain_field() { let error = trace.error(); assert!( matches!( - error.inner(), + error, Error::CallDepthExceeded { cap: 8, via: DepthVia::Fanout @@ -218,7 +218,7 @@ fn two_arms_writing_one_path_terminate_the_run_with_a_determinism_violation() { ```\n"; let trace = scheduler_scenario(md).run(); - match trace.error().inner() { + match trace.error() { Error::Determinism(detail) => { assert!(detail.contains("shared.txt"), "error was: {detail}"); assert!( @@ -275,7 +275,7 @@ fn two_live_arms_appending_one_path_terminate_with_a_determinism_violation() { ```\n"; let trace = scheduler_scenario(md).run(); - match trace.error().inner() { + match trace.error() { Error::Determinism(detail) => { assert!(detail.contains("evidence.md"), "error was: {detail}"); assert_eq!( diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/fanout_model.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/fanout_model.rs new file mode 100644 index 000000000..568972968 --- /dev/null +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/fanout_model.rs @@ -0,0 +1,200 @@ +//! The model-first `tasks.fanout(model, worker, collection)`: every arm runs +//! on the given model, the parent's own model is what it was before the call +//! (also when a spawn fails part-way), a raw-id handle works with no model +//! supplied, a first argument that is not a handle and a wrong argument count +//! are refused, and an arm's own switch stays inside that arm. + +use super::*; + +#[test] +fn a_three_argument_fanout_runs_every_arm_on_the_given_model() { + let md = sections(&[ + ( + "## Main", + "local r = tasks.fanout(models.get('fast'), '### Arm', {'a', 'b', 'c'})\n\ + return table.concat(r, ',')", + ), + ("### Arm", "return models.current().model_id"), + ]); + let trace = chain_scenario(&md).limits(ceiling(4)).run(); + assert_eq!(trace.text(), "model-c,model-c,model-c"); +} + +#[test] +fn the_parents_model_after_a_three_argument_fanout_is_what_it_was_before() { + let md = sections(&[ + ( + "## Main", + "models.use('strong')\n\ + local r = tasks.fanout(models.get('fast'), '### Arm', {'a', 'b'})\n\ + return table.concat(r, ',') .. '|' .. models.current().label", + ), + ("### Arm", "return models.current().label"), + ]); + let trace = chain_scenario(&md).limits(ceiling(4)).run(); + assert_eq!(trace.text(), "fast,fast|strong"); +} + +#[test] +fn a_spawn_failure_part_way_still_restores_the_parents_model() { + // The chain at depth 8 asks a model-first fanout, whose spawn the depth + // cap refuses. The shim restores the chain's model before it re-raises, + // so the label read after the caught failure is the label read before it. + let md = sections(&[ + ( + "## Alpha", + "var.n = (var.n or 0) + 1\n\ + if var.n == 1 then models.use('strong') end\n\ + if var.n >= 9 then\n\ + local before = models.current().label\n\ + local ok, err = pcall(tasks.fanout, models.get('fast'), '### Worker', {'x'})\n\ + local cap = tostring(err):match('recursion exceeded cap of %d+')\n\ + return before .. '|' .. tostring(ok) .. '|' .. tostring(cap) .. '|' .. models.current().label\n\ + end\n\ + return call('## Beta')", + ), + ("### Worker", "return item"), + ("## Beta", "var.n = var.n + 1\nreturn call('## Alpha')"), + ]); + let trace = chain_scenario(&md).run(); + assert_eq!( + trace.text(), + "strong|false|recursion exceeded cap of 8|strong" + ); +} + +#[test] +fn a_raw_id_handle_fans_out_with_no_model_supplied() { + let md = sections(&[ + ( + "## Main", + "local r = tasks.fanout(models.get('vendor/raw-model'), '### Arm', {'a', 'b'})\n\ + return table.concat(r, ',')", + ), + ("### Arm", "return models.current().model_id"), + ]); + let trace = Scenario::new(md) + .models(ModelSet::default()) + .limits(ceiling(4)) + .config(|ctx| ctx.with_ui(serde_json::json!({}))) + .run(); + assert_eq!(trace.text(), "vendor/raw-model,vendor/raw-model"); +} + +#[test] +fn a_failing_arm_still_leaves_the_parents_model_as_it_was_before_the_call() { + // The arm raises, so the fanout re-raises the failure at its join. The + // model is already back by then, so the label read after the caught + // failure is the label read before the call. + let md = sections(&[ + ( + "## Main", + "models.use('strong')\n\ + local before = models.current().label\n\ + local ok = pcall(tasks.fanout, models.get('fast'), '### Arm', {'a'})\n\ + return before .. '|' .. tostring(ok) .. '|' .. models.current().label", + ), + ("### Arm", "error('boom')"), + ]); + let trace = chain_scenario(&md).limits(ceiling(4)).run(); + assert_eq!(trace.text(), "strong|false|strong"); +} + +#[test] +fn a_first_argument_that_is_not_a_model_handle_is_refused() { + let md = sections(&[ + ( + "## Main", + "local out = {}\n\ + for _, first in ipairs({ 'fast', 5 }) do\n\ + local ok, err = pcall(tasks.fanout, first, '### Arm', {'a'})\n\ + assert(not ok, 'a non-handle first argument must be refused')\n\ + out[#out + 1] = tostring(err)\n\ + end\n\ + local ok, err = pcall(tasks.fanout, nil, '### Arm', {'a'})\n\ + assert(not ok, 'a nil first argument must be refused')\n\ + out[#out + 1] = tostring(err)\n\ + return table.concat(out, '\\n')", + ), + ("### Arm", "return item"), + ]); + let trace = chain_scenario(&md).run(); + let text = trace.text(); + for got in ["string", "number", "nil"] { + assert!( + text.contains(&format!( + "tasks.fanout: the first argument must be a model handle, got {got}" + )), + "the refusal names the {got} first argument: {text}" + ); + } +} + +#[test] +fn an_argument_count_other_than_two_or_three_is_refused() { + let md = sections(&[( + "## Main", + "local ok0, err0 = pcall(tasks.fanout)\n\ + local ok1, err1 = pcall(tasks.fanout, '### Arm')\n\ + local ok4, err4 = pcall(tasks.fanout, models.get('fast'), '### Arm', {'a'}, 4)\n\ + assert(not ok0 and not ok1 and not ok4, 'a wrong argument count must be refused')\n\ + return tostring(err0) .. '\\n' .. tostring(err1) .. '\\n' .. tostring(err4)", + )]); + let trace = chain_scenario(&md).run(); + let text = trace.text(); + assert_eq!( + text.matches("tasks.fanout takes (worker, collection) or (model, worker, collection)") + .count(), + 3, + "each wrong count is refused with the same text: {text}" + ); +} + +#[test] +fn an_arms_own_switch_stays_inside_that_arm() { + let md = sections(&[ + ( + "## Main", + "local r = tasks.fanout(models.get('fast'), '### Arm', {'a', 'b'})\n\ + return table.concat(r, ',') .. '|' .. models.current().label", + ), + ( + "### Arm", + "if item == 'a' then models.use('strong') end\nreturn models.current().label", + ), + ]); + let trace = chain_scenario(&md).limits(ceiling(4)).run(); + assert_eq!(trace.text(), "strong,fast|default"); +} + +#[test] +fn models_use_default_inside_an_arm_selects_the_default_role() { + let md = sections(&[ + ( + "## Main", + "local r = tasks.fanout(models.get('fast'), '### Arm', {'a', 'b'})\n\ + return table.concat(r, ',')", + ), + ( + "### Arm", + "models.use('default')\nreturn models.current().label", + ), + ]); + let trace = chain_scenario(&md).limits(ceiling(4)).run(); + assert_eq!(trace.text(), "default,default"); +} + +#[test] +fn a_two_argument_fanout_gives_every_arm_the_parents_current_model() { + let md = sections(&[ + ( + "## Main", + "models.use('strong')\n\ + local r = tasks.fanout('### Arm', {'a', 'b'})\n\ + return table.concat(r, ',')", + ), + ("### Arm", "return models.current().label"), + ]); + let trace = chain_scenario(&md).limits(ceiling(4)).run(); + assert_eq!(trace.text(), "strong,strong"); +} diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/h1.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/h1.rs index 757e05bcb..58f86b7b3 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler/h1.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/h1.rs @@ -3,12 +3,10 @@ use super::*; -/// A run of `source` for a scheduler H1 test: unprepared, with every -/// declared role bound to the test model the way prepare's trivial fill -/// binds it, and the shared model set empty - the H1 pass under test -/// records its own bindings. +/// A run of `source` for a scheduler H1 test: under the test model, which +/// fills all five roles. fn h1_scenario(source: &str) -> Scenario { - Scenario::new(source).unprepared().bind_roles() + Scenario::new(source).bound() } mod control_flow; diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/h1/control_flow.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/h1/control_flow.rs index 76f0553d8..2a2a5f101 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler/h1/control_flow.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/h1/control_flow.rs @@ -124,10 +124,9 @@ fn the_h1_decision_tool_idiom_runs_before_the_walk() { calls(&[("call_1", "decide", "{\"choice\":\"use_mcp\"}")]), text("decided"), ]); - let md = "---\nname: h1-decision\ndescription: d\nrunvil: 0\nmodels:\n writer: {}\n---\n\n\ + let md = "---\nname: h1-decision\ndescription: d\nrunvil: 0\n---\n\n\ # Decide\n\n\ ```lua\n\ - models.default('writer')\n\ tools.offer(tools.adhoc({ name = 'decide', description = 'Record the verdict', parameters = { choice = 'string' } }, function(args)\n\ var.verdict = args.choice\n\ return 'recorded'\n\ diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/h1/pass.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/h1/pass.rs index c709fa35c..bf547cf07 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler/h1/pass.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/h1/pass.rs @@ -12,11 +12,11 @@ fn h1_infer_runs_once() { // The H1 pass selects the default model by label, a handle's `infer` // yields through the shim, and the H1 `var` hand-off seeds the walk. let gateway = ScriptedChat::new(vec![text("h1 answer")]); - let md = "---\nname: live-h1\ndescription: d\nrunvil: 0\nmodels:\n writer: {}\n---\n\n\ + let md = "---\nname: live-h1\ndescription: d\nrunvil: 0\n---\n\n\ # Live H1\n\n\ ```lua\n\ - local writer = models.default('writer')\n\ - var.answer = writer:infer('answer once')\n\ + local writer = models.get('default')\n\ + var.answer = models.infer(writer, 'answer once')\n\ ```\n\n\ ## Result\n\n\ ```lua\nreturn var.answer\n```\n"; @@ -32,10 +32,9 @@ fn h1_models_infer_resolves_the_default_model_without_touching_sys() { // shared set and runs the one infer shape - a single tool-free // round on a fresh conversation that leaves `sys` untouched. let gateway = ScriptedChat::new(vec![text("h1 answer")]); - let md = "---\nname: live-h1-models-infer\ndescription: d\nrunvil: 0\nmodels:\n writer: {}\n---\n\n\ + let md = "---\nname: live-h1-models-infer\ndescription: d\nrunvil: 0\n---\n\n\ # Live H1 Models Infer\n\n\ ```lua\n\ - models.default('writer')\n\ var.answer = models.infer('answer once')\n\ var.sys_untouched = not pcall(function() return sys.reply_finish_reason end)\n\ ```\n\n\ @@ -99,7 +98,7 @@ fn a_failed_h1_assertion_ends_the_run_as_requirements_unmet() { let trace = h1_scenario(md).run(); let error = trace.error(); - let notice = match error.inner() { + let notice = match error { Error::RequirementsUnmet { notice, .. } => notice, other => panic!("the failed assertion classifies as RequirementsUnmet: {other:?}"), }; @@ -136,7 +135,7 @@ fn an_uncaught_h1_assertion_reports_the_chunk_failed() { let error = trace.error(); assert!( - matches!(error.inner(), Error::RequirementsUnmet { .. }), + matches!(error, Error::RequirementsUnmet { .. }), "the failed gate classifies as RequirementsUnmet: {error}" ); let observed = sequence(&trace); @@ -162,7 +161,7 @@ fn an_h1_scalar_return_still_reads_var_back() { let error = trace.error(); assert!( - matches!(error.inner(), Error::Lua { cause: None, .. }), + matches!(error, Error::Lua { cause: None, .. }), "the read-back failure is machinery around the prompt's chunk, \ not the failed gate, so it keeps the Lua kind: {error}" ); @@ -191,7 +190,7 @@ fn a_failing_library_required_from_h1_ends_the_run_as_requirements_unmet() { let error = trace.error(); assert!( - matches!(error.inner(), Error::RequirementsUnmet { .. }), + matches!(error, Error::RequirementsUnmet { .. }), "the library's raise is the failed H1 gate: {error}" ); let cause = error diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/h1/returns_and_prose.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/h1/returns_and_prose.rs index 1396e8d15..0d9209d3f 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler/h1/returns_and_prose.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/h1/returns_and_prose.rs @@ -48,10 +48,9 @@ fn h1_prose_inferred_explicitly_is_the_run_result() { // infer ends the run with the inferred text: the scalar return // short-circuits the (empty) walk. let gateway = ScriptedChat::new(vec![text("h1 reply")]); - let md = "---\nname: t\ndescription: d\nrunvil: 0\nmodels:\n writer: {}\n---\n\n\ + let md = "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n\ # Only Prose\n\n\ ```lua\n\ - models.default('writer')\n\ ```\n\n\ say something\n\n\ ```lua\n\ @@ -68,10 +67,9 @@ fn h1_and_h2_prose_each_infer_explicitly_in_source_order() { // The H1 pass and the H2 section each read their own pending // buffer into an explicit infer - two completions, in source order. let gateway = ScriptedChat::new(vec![text("h1 reply"), text("h2 reply")]); - let md = "---\nname: shared-loop\ndescription: d\nrunvil: 0\nmodels:\n writer: {}\n---\n\n\ + let md = "---\nname: shared-loop\ndescription: d\nrunvil: 0\n---\n\n\ # Shared Loop\n\n\ ```lua\n\ - models.default('writer')\n\ ```\n\n\ h1 prose turn\n\n\ ```lua\n\ @@ -115,16 +113,16 @@ fn unread_h1_prose_stays_inert_and_explicit_infer_requires_a_model() { {{ var.omit }}\n\n\ ## Result\n\n\ ```lua\nreturn 'ok'\n```\n"; - assert_eq!(h1_scenario(unread).run().text(), "ok"); + assert_eq!(Scenario::new(unread).run().text(), "ok"); let reading = "---\nname: read-h1\ndescription: d\nrunvil: 0\n---\n\n\ # Read H1\n\n\ ask\n\n\ ```lua\nreturn models.infer(prose)\n```\n"; - let trace = h1_scenario(reading).run(); + let trace = Scenario::new(reading).run(); let error = trace.error(); assert!( - matches!(error.inner(), Error::ModelRequired { .. }), + matches!(error, Error::ModelRequired { .. }), "expected ModelRequired, got {error}" ); } @@ -134,10 +132,9 @@ fn h1_prose_infers_explicitly_and_var_accumulates_into_the_walk() { // The pass reads its pending buffer only through an explicit infer, and // `var` writes accumulate across the pass into the walk. let gateway = ScriptedChat::new(vec![text("final answer")]); - let md = "---\nname: live-h1-prose\ndescription: d\nrunvil: 0\nmodels:\n writer: {}\n---\n\n\ + let md = "---\nname: live-h1-prose\ndescription: d\nrunvil: 0\n---\n\n\ # Live H1 Prose\n\n\ ```lua\n\ - models.default('writer')\n\ var.executions = (var.executions or 0) + 1\n\ ```\n\n\ Ask for one round.\n\n\ diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/walk/chains.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/walk/chains.rs index 7a396be11..4c0c94e29 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler/walk/chains.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/walk/chains.rs @@ -252,7 +252,7 @@ fn a_jump_descent_does_not_consume_call_depth() { let error = rig.trace().error(); assert!( matches!( - error.inner(), + error, Error::CallDepthExceeded { cap: 8, via: DepthVia::Call diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/walk/driving.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/walk/driving.rs index da5760ebb..5b79ad0bd 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler/walk/driving.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/walk/driving.rs @@ -59,9 +59,9 @@ fn cancellation_while_suspended_on_infer_interrupts_the_run() { .run(); assert!( - matches!(trace.result(), RunResult::Cancelled), + matches!(trace.outcome(), RunOutcome::Cancelled), "cancelling a suspended infer must interrupt the run, got {:?}", - trace.result() + trace.outcome() ); assert!( matches!( @@ -90,7 +90,7 @@ fn call_depth_cap_reads_the_chain_field() { let error = trace.error(); assert!( matches!( - error.inner(), + error, Error::CallDepthExceeded { cap: 8, via: DepthVia::Call diff --git a/crates/runvil-internal/engine/src/execute/tests/scheduler/walk/jumps.rs b/crates/runvil-internal/engine/src/execute/tests/scheduler/walk/jumps.rs index 010c19f51..cb57d0a47 100644 --- a/crates/runvil-internal/engine/src/execute/tests/scheduler/walk/jumps.rs +++ b/crates/runvil-internal/engine/src/execute/tests/scheduler/walk/jumps.rs @@ -136,9 +136,9 @@ fn an_erroring_section_reports_started_but_not_finished() { let trace = scheduler_scenario(md).run(); assert!( - matches!(trace.result(), RunResult::Failure(_)), + matches!(trace.outcome(), RunOutcome::Failed { .. }), "the erroring section fails the run, got {:?}", - trace.result() + trace.outcome() ); let observed = sequence(&trace); assert!( diff --git a/crates/runvil-internal/engine/src/execute/tests/section_messages.rs b/crates/runvil-internal/engine/src/execute/tests/section_messages.rs index 93d3a2d70..8cdaa91aa 100644 --- a/crates/runvil-internal/engine/src/execute/tests/section_messages.rs +++ b/crates/runvil-internal/engine/src/execute/tests/section_messages.rs @@ -310,27 +310,26 @@ fn meta_reaches_a_tool_through_a_view_but_never_the_model() { } #[test] -fn both_loop_entries_take_only_a_compactor_and_check_it_before_a_running_loop() { +fn the_loop_entry_takes_its_two_shapes_and_checks_the_compactor_before_a_running_loop() { let source = PromptSource::new().section( "Only", - "local other = models.get('other') + "local other = models.get('strong') local refusals = {} local function refuse(call) local ok, err = pcall(call) refusals[#refusals + 1] = tostring(ok) .. ':' .. err.kind .. ':' .. tostring(err) end - tools.offer(tools.adhoc({ name = 'probe', description = 'Probe the loop entries', parameters = { value = 'string' } }, function() - refuse(function() return models.loop(other) end) + tools.offer(tools.adhoc({ name = 'probe', description = 'Probe the loop entry', parameters = { value = 'string' } }, function() refuse(function() return models.loop(nil, nil) end) + refuse(function() return models.loop(other, nil, nil) end) refuse(function() return models.loop(42) end) refuse(function() return models.loop() end) - refuse(function() return other.loop() end) - refuse(function() return other:loop(nil, nil) end) - refuse(function() return other:loop(42) end) - refuse(function() return other:loop(compactors.fail) end) + refuse(function() return models.loop(other) end) + refuse(function() return models.loop(other, 42) end) + refuse(function() return models.loop(compactors.fail) end) return 'probed' end)) - messages:user('probe the entries') + messages:user('probe the entry') models.loop() return table.concat(refusals, '\\n') .. '\\n' .. #messages", ); @@ -338,22 +337,22 @@ fn both_loop_entries_take_only_a_compactor_and_check_it_before_a_running_loop() &source, vec![calls(&[("c1", "probe", "{\"value\":\"x\"}")]), text("done")], ); + let arity = "models.loop takes (compactor?) or (model, compactor?)"; let compactor = "compactor must be a function, got integer"; let expected = [ - "models.loop takes (compactor?); call handle:loop(compactor?) to run on a model handle", - "models.loop takes (compactor?)", + arity, + arity, compactor, LOOP_RUNNING, - "call loop on a model handle with a colon: handle:loop(compactor?)", - "handle:loop takes (compactor?)", + LOOP_RUNNING, compactor, LOOP_RUNNING, ] .map(|message| format!("false:lua:{message}")); let lines: Vec<&str> = out.lines().collect(); - assert_eq!(lines[..8], expected); + assert_eq!(lines[..7], expected); assert_eq!( - lines[8], "4", + lines[7], "4", "the user message, the call, its result, and the reply" ); assert_eq!(gateway.call_count(), 2, "no refused entry sends a round"); diff --git a/crates/runvil-internal/engine/src/execute/tests/serial_driver.rs b/crates/runvil-internal/engine/src/execute/tests/serial_driver.rs index 006e99802..baf34ac1a 100644 --- a/crates/runvil-internal/engine/src/execute/tests/serial_driver.rs +++ b/crates/runvil-internal/engine/src/execute/tests/serial_driver.rs @@ -1,7 +1,7 @@ //! The serial sans-IO driver over `Run`, and the properties it makes //! testable without a runtime or a gateway: a literal prompt that issues no //! effect, a three-arm fanout whose answers arrive in reverse order, -//! determinism (two runs under the same context and answers produce +//! determinism (two runs under the same config and answers produce //! identical effects, events, provenances, and `sys.id`s), the //! batching-pairing property (answers delivered one per step, all at once, //! and shuffled within a batch produce identical per-task effects and @@ -9,7 +9,7 @@ use runvil_types::event::Event; -use super::context::{parse, test_context}; +use super::context::{parse, test_config}; use super::scheduler::scheduler_scenario; use super::*; use crate::execute::run::{Effect, Run, Step}; @@ -56,9 +56,9 @@ fn the_driver_runs_a_literal_prompt_with_no_effect() { // A literal return issues nothing, so the performer never runs, and the // events open and close with the run's boundaries. let md = "---\nname: t\ndescription: d\nrunvil: 0\n---\n\n# Title\n\n## Only\n\n```lua\nreturn 'hello'\n```\n"; - let run = Run::new(Arc::new(parse(md)), "", test_context(EXECUTION)); - let (result, events) = drive(run, |_, effect| panic!("no effect is issued: {effect:?}")); - assert_eq!(text_of(result), "hello"); + let run = Run::new(Arc::new(parse(md)), "", test_config(EXECUTION)); + let (outcome, events) = drive(run, |_, effect| panic!("no effect is issued: {effect:?}")); + assert_eq!(text_of(outcome), "hello"); assert!( matches!(events.first(), Some(Event::RunStarted { .. })), "the events open with the run's start: {events:?}" @@ -78,10 +78,10 @@ fn the_driver_performs_store_and_model_effects_and_keeps_the_events_in_order() { store.write('notes.md', 'kept')\n\ return store.read('notes.md') .. '/' .. models.infer('ask')\n\ ```\n"; - let (result, events) = drive(model_run(md), |_, effect| { + let (outcome, events) = drive(model_run(md), |_, effect| { perform_locally(effect, &mut echo_chat) }); - assert_eq!(text_of(result), "kept/r(ask)"); + assert_eq!(text_of(outcome), "kept/r(ask)"); let order: Vec<&str> = events .iter() .filter_map(|event| match event { @@ -123,13 +123,13 @@ fn a_three_arm_fanout_fed_its_answers_in_reverse_order_packs_results_in_collecti for issued in effects.into_iter().rev() { run.resume(issued.id, echo_chat(&issued.effect)); } - let (result, events) = match run.step() { - Step::Done { result, events } => (result, events), + let (outcome, events) = match run.step() { + Step::Done { outcome, events } => (outcome, events), other @ Step::Pending { .. } => { panic!("every arm is answered, so the fanout completes: {other:?}") } }; - assert_eq!(text_of(result), "0.1|0.0.0=r(a)|0.1.0=r(b)|0.2.0=r(c)"); + assert_eq!(text_of(outcome), "0.1|0.0.0=r(a)|0.1.0=r(b)|0.2.0=r(c)"); assert_eq!( succeeded(&events), ["0.2", "0.1", "0.0"], @@ -138,8 +138,8 @@ fn a_three_arm_fanout_fed_its_answers_in_reverse_order_packs_results_in_collecti } #[test] -fn two_runs_under_the_same_context_and_answers_are_identical() { - // The determinism property: the same prompt, context, and answers +fn two_runs_under_the_same_config_and_answers_are_identical() { + // The determinism property: the same prompt, config, and answers // produce the same text (the `sys.id`s in it), the same events in the // same order with the same provenances, and the same effects. let first = drive_batched(model_run(FANOUT), Batching::AllAtOnce); diff --git a/crates/runvil-internal/engine/src/execute/tests/suite.rs b/crates/runvil-internal/engine/src/execute/tests/suite.rs index d0bd0e215..85a9336bb 100644 --- a/crates/runvil-internal/engine/src/execute/tests/suite.rs +++ b/crates/runvil-internal/engine/src/execute/tests/suite.rs @@ -2,8 +2,9 @@ //! author writes them and drives their execution on the serial test driver //! without a live gateway. Split by domain - parsing contracts, section //! execution, fanout, control flow, the args/argv and lazy-prose surfaces, -//! an author's own `_G` metatable, `require`, and the prepare pass - over -//! the run-level [`Scenario`](crate::test_support::scenario::Scenario). Its +//! an author's own `_G` metatable, `require`, and the role map and start +//! checks - over the run-level +//! [`Scenario`](crate::test_support::scenario::Scenario). Its //! cases reach engine-only items (the test drivers and recorders, the store //! facade, the parser's Lua programs), so they run here rather than against //! the `runvil` facade, whose own suite holds the rest. diff --git a/crates/runvil-internal/engine/src/execute/tests/suite/exec_flow/run_setup.rs b/crates/runvil-internal/engine/src/execute/tests/suite/exec_flow/run_setup.rs index d3378eef5..b2402cb11 100644 --- a/crates/runvil-internal/engine/src/execute/tests/suite/exec_flow/run_setup.rs +++ b/crates/runvil-internal/engine/src/execute/tests/suite/exec_flow/run_setup.rs @@ -1,4 +1,4 @@ -//! What a run starts with: its store handle, a default context, and the +//! What a run starts with: its store handle, a default config, and the //! tool and model bindings its frontmatter declares. use std::sync::Arc; @@ -6,21 +6,21 @@ use std::sync::Arc; use super::*; use crate::RunErrorKind; -use crate::RunResult; -use crate::execute::tests::context::test_context; +use crate::RunOutcome; +use crate::execute::tests::context::test_config; use crate::execute::tests::fixtures::echo_tool; use crate::test_support::scenario::EXECUTION; use runvil_vfs::{VfsError, VfsRef}; /// A hand-built `VfsRef` that declares no store fails the run up front /// with [`RunErrorKind::Vfs`], saying what is missing: every run needs a -/// declared store. (`RunContext::prepare` never replaces the handle, so -/// the raw handle's declaration is what the run sees.) +/// declared store. (Nothing replaces the handle, so the raw handle's +/// declaration is what the run sees.) #[test] fn a_handle_without_a_declared_store_fails_the_run() { let source = one_section("store.write('overlay.txt', 'overlaid')"); let vfs = VfsRef::new(runvil_vfs::MemoryBackend::new()); - let trace = Scenario::new(source).vfs(vfs).unprepared().run(); + let trace = Scenario::new(source).vfs(vfs).run(); let error = trace.error(); assert_eq!( error.kind(), @@ -59,28 +59,28 @@ fn a_store_backend_that_refuses_its_session_fails_the_run_at_the_first_step() { let mut run = crate::execute::run::Run::new( Arc::new(prompt.expect("the fixture parses")), "", - test_context(EXECUTION).with_vfs(vfs), + test_config(EXECUTION).with_vfs(vfs), ); match run.step() { crate::execute::run::Step::Done { - result: RunResult::Failure(error), + outcome: RunOutcome::Failed { kind, message }, .. - } => assert_eq!(error.kind(), RunErrorKind::Vfs, "{error}"), + } => assert_eq!(kind, RunErrorKind::Vfs, "{message}"), other => panic!("expected the first step to fail the run, got {other:?}"), } } #[test] -fn a_default_context_runs_a_pack_free_prompt() { - // A prompt declaring no Packs runs under a default context: no +fn a_default_config_runs_a_pack_free_prompt() { + // A prompt declaring no Packs runs under a default config: no // catalog, no roster, no client, no real roots. let trace = Scenario::new(one_section("return 'no Packs'")).run(); assert_eq!(trace.text(), "no Packs"); } #[test] -fn default_run_context_store_handle_declares_a_fresh_store() { - // `RunContext` absorbs the filesystem handle with a `VfsRef::default()` +fn default_run_config_store_handle_declares_a_fresh_store() { + // `RunConfig` absorbs the filesystem handle with a `VfsRef::default()` // (a fresh memory store at `/`) default: a store-using run needs no // caller-supplied handle. let source = prompt( diff --git a/crates/runvil-internal/engine/src/execute/tests/suite/exec_flow/store_failures.rs b/crates/runvil-internal/engine/src/execute/tests/suite/exec_flow/store_failures.rs index 27523e92a..bfec06beb 100644 --- a/crates/runvil-internal/engine/src/execute/tests/suite/exec_flow/store_failures.rs +++ b/crates/runvil-internal/engine/src/execute/tests/suite/exec_flow/store_failures.rs @@ -6,7 +6,9 @@ use runvil_lua::VfsOp; use super::*; -use crate::{EffectRecord, RunErrorKind}; +use crate::Error; +use crate::RunErrorKind; +use crate::execute::EffectRecord; /// A store failure the author caught with `pcall` is an error table of /// kind `store`: its `reason`, `path`, and `anchor` fields and its @@ -142,7 +144,7 @@ fn a_store_at_the_root_cannot_reach_a_mount_beneath_it() { .expect("the handle acquires") .write("/mount/secret.txt", b"secret") .expect("the mount seeds"); - let trace = Scenario::new(source).vfs(vfs).unprepared().run(); + let trace = Scenario::new(source).vfs(vfs).run(); assert_eq!( trace.text(), "ok", @@ -181,11 +183,11 @@ fn a_library_write_across_two_arms_ends_the_run_with_determinism() { let source = arms_requiring_a_seeder("store.write('seeded.txt', 'seed')"); let trace = Scenario::new(source).run(); let error = trace.error(); - assert_eq!( - error.kind(), - RunErrorKind::Determinism, + assert!( + matches!(error, Error::Determinism(_)), "the load-time conflict is a determinism violation: {error:?}" ); + assert_eq!(error.kind(), RunErrorKind::Vfs, "{error:?}"); } /// A conflict a library caught with its own `pcall` while loading still @@ -197,11 +199,11 @@ fn a_library_conflict_caught_with_pcall_still_ends_the_run_with_determinism() { ); let trace = Scenario::new(source).run(); let error = trace.error(); - assert_eq!( - error.kind(), - RunErrorKind::Determinism, + assert!( + matches!(error, Error::Determinism(_)), "a caught load-time conflict is still fatal: {error:?}" ); + assert_eq!(error.kind(), RunErrorKind::Vfs, "{error:?}"); } /// The library the captured-store cases require: `store.write` captured @@ -239,11 +241,11 @@ fn a_captured_store_function_conflict_caught_with_pcall_still_ends_the_run_with_ ); let trace = Scenario::new(source).run(); let error = trace.error(); - assert_eq!( - error.kind(), - RunErrorKind::Determinism, + assert!( + matches!(error, Error::Determinism(_)), "a conflict through a captured store function is fatal: {error:?}" ); + assert_eq!(error.kind(), RunErrorKind::Vfs, "{error:?}"); } /// A store function a library captured at load time reaches the caller diff --git a/crates/runvil-internal/engine/src/execute/tests/suite/fanout.rs b/crates/runvil-internal/engine/src/execute/tests/suite/fanout.rs index 02818e13b..168849230 100644 --- a/crates/runvil-internal/engine/src/execute/tests/suite/fanout.rs +++ b/crates/runvil-internal/engine/src/execute/tests/suite/fanout.rs @@ -3,6 +3,7 @@ use runvil_parser::test_support::prompt; +use crate::Error; use crate::RunErrorKind; use crate::test_support::scenario::{Record, Scenario}; @@ -120,10 +121,14 @@ fn a_cross_arm_append_terminates_the_run_with_a_determinism_violation() { .name(FANOUT_CROSS_ARM_EXECUTION) .run(); let error = trace.error(); + assert!( + matches!(error, Error::Determinism(_)), + "a conflicting access is a determinism violation: {error:?}" + ); assert_eq!( error.kind(), - RunErrorKind::Determinism, - "a conflicting access classifies as a determinism violation: {error:?}" + RunErrorKind::Vfs, + "a determinism violation classifies as a store failure: {error:?}" ); let text = error.to_string(); assert!( diff --git a/crates/runvil-internal/engine/src/execute/tests/suite/prepare-refusals.rs b/crates/runvil-internal/engine/src/execute/tests/suite/prepare-refusals.rs new file mode 100644 index 000000000..291f28c05 --- /dev/null +++ b/crates/runvil-internal/engine/src/execute/tests/suite/prepare-refusals.rs @@ -0,0 +1,208 @@ +//! The run's first step refusing an unsatisfiable prompt with the +//! model-readable notice, and running a satisfiable one; the version check +//! running before the requirements; and a refusal after the run's cancel +//! fired ending the run cancelled. + +use runvil_types::cancel::CancelHandle; +use runvil_types::event::Event; +use runvil_types::models::ThinkingMode; + +use super::{current_model, refusal, under_model}; +use crate::Error; +use crate::RunErrorKind; +use crate::execute::RunOutcome; +use crate::test_support::scenario::Scenario; +use crate::test_support::scripted_chat::{ScriptedChat, text}; + +/// A prompt declaring one model role with a thinking requirement and a +/// context minimum. +const DECLARES_STRONG: &str = concat!( + "---\n", + "name: declares-strong\n", + "description: d\n", + "runvil: 0\n", + "models:\n", + " strong:\n", + " thinking: true\n", + " min_context: 200000\n", + "---\n\n", + "# Title\n\n", + "## Only\n\n", + "```lua\n", + "return 'done'\n", + "```\n", +); + +#[test] +fn the_first_step_refuses_an_unsatisfiable_prompt_with_a_model_readable_notice() { + let model = current_model(32_000, ThinkingMode::Never); + let notice = refusal(under_model(DECLARES_STRONG, "refuse", model)); + // The notice is written to be read by a model: it names the role, + // each failed check, and required versus actual. + assert!( + notice.starts_with("the environment cannot satisfy this prompt:"), + "the notice opens with the standing refusal line: {notice}" + ); + assert!( + notice.contains( + "- role 'strong': requires a context of at least 200000 tokens; \ + the model filling it provides 32000" + ), + "the notice gives required versus actual context: {notice}" + ); + assert!( + notice.contains( + "- role 'strong': requires 'thinking: true'; \ + the thinking capability of the model filling it is Never" + ), + "the notice gives required versus actual thinking: {notice}" + ); +} + +#[test] +fn a_run_whose_role_map_meets_the_prompt_runs() { + // The declared role's requirements are met by the model that fills it. + let model = current_model(200_000, ThinkingMode::Always); + let trace = under_model(DECLARES_STRONG, "satisfied", model).run(); + assert_eq!(trace.text(), "done"); +} + +/// A prompt declaring one role that requires no thinking, whose section +/// asks it one question. +const DECLARES_NO_THINKING: &str = concat!( + "---\n", + "name: declares-no-thinking\n", + "description: d\n", + "runvil: 0\n", + "models:\n", + " small:\n", + " thinking: false\n", + "---\n\n", + "# Title\n\n", + "## Only\n\n", + "```lua\n", + "models.use('small')\n", + "return models.infer('ping')\n", + "```\n", +); + +#[test] +fn a_no_thinking_role_on_a_switchable_model_runs_and_asks_for_thinking_off() { + let gateway = ScriptedChat::new(vec![text("pong")]); + let model = current_model(32_000, ThinkingMode::Switchable); + let trace = under_model(DECLARES_NO_THINKING, "switchable", model) + .chat(&gateway) + .run(); + // An unmet requirement would refuse the run, so the reply also shows + // that a switchable model can turn thinking off. + assert_eq!(trace.text(), "pong"); + let body = gateway + .last_request() + .expect("the round reaches the gateway"); + assert_eq!(body.options.model(), "current"); + assert_eq!(body.options.thinking(), Some(false)); +} + +#[test] +fn a_no_thinking_role_on_an_always_thinking_model_is_refused() { + let model = current_model(32_000, ThinkingMode::Always); + let notice = refusal(under_model(DECLARES_NO_THINKING, "always", model)); + assert!( + notice.contains( + "- role 'small': requires 'thinking: false'; \ + the thinking capability of the model filling it is Always" + ), + "the notice gives required versus actual thinking: {notice}" + ); +} + +/// A prompt declaring one role that needs a long context. +const DECLARES_LONG_CONTEXT: &str = concat!( + "---\n", + "name: declares-long-context\n", + "description: d\n", + "runvil: 0\n", + "models:\n", + " creative:\n", + " min_context: 200000\n", + "---\n\n", + "# Title\n\n", + "## Only\n\n", + "Done.\n", +); + +/// An unmet requirement refuses the run with the model-readable notice +/// text, line for line. +#[test] +fn an_unmet_requirement_produces_the_model_readable_notice() { + let model = current_model(32_000, ThinkingMode::Switchable); + assert_eq!( + refusal(under_model(DECLARES_LONG_CONTEXT, "notice", model)), + "the environment cannot satisfy this prompt:\n\ + - role 'creative': requires a context of at least 200000 tokens; \ + the model filling it provides 32000" + ); +} + +#[test] +fn an_unsupported_version_is_reported_before_an_unmet_requirement() { + let source = DECLARES_LONG_CONTEXT.replace("runvil: 0", "runvil: 7"); + let model = current_model(32_000, ThinkingMode::Switchable); + let trace = under_model(&source, "version-first", model).run(); + let error = trace.error(); + assert!( + matches!(error, Error::UnsupportedVersion(7)), + "the version gate refuses the run first: {error}" + ); + assert_eq!(error.kind(), RunErrorKind::Parse, "{error}"); + assert!( + trace.run_events().is_empty(), + "nothing ran, nothing reported" + ); +} + +/// `scenario`, with its run's cancel flag set before the run is built. +fn cancelled_before_it_starts(scenario: Scenario) -> Scenario { + let cancel = CancelHandle::new(); + cancel.cancel(); + scenario.config(move |ctx| ctx.with_cancel(cancel)) +} + +#[test] +fn a_refusal_after_the_runs_cancel_fired_ends_the_run_cancelled_with_no_run_events() { + let model = current_model(32_000, ThinkingMode::Switchable); + let unsupported = DECLARES_LONG_CONTEXT.replace("runvil: 0", "runvil: 7"); + for (source, refused_for) in [ + (DECLARES_LONG_CONTEXT, "an unmet requirement"), + (unsupported.as_str(), "an unsupported version"), + ] { + let trace = + cancelled_before_it_starts(under_model(source, "cancelled", model.clone())).run(); + assert!( + matches!(trace.outcome(), RunOutcome::Cancelled), + "{refused_for} after a cancel ends the run cancelled, got {:?}", + trace.outcome() + ); + assert!( + trace.run_events().is_empty(), + "{refused_for}: a run that never started reports no run events: {:?}", + trace.run_events() + ); + } +} + +#[test] +fn a_run_that_passes_its_start_checks_after_a_cancel_starts_and_ends_cancelled() { + let model = current_model(200_000, ThinkingMode::Always); + let trace = cancelled_before_it_starts(under_model(DECLARES_STRONG, "started", model)).run(); + assert!( + matches!(trace.outcome(), RunOutcome::Cancelled), + "got {:?}", + trace.outcome() + ); + assert!( + matches!(trace.run_events().first(), Some(Event::RunStarted { .. })), + "the run started before it saw the cancel: {:?}", + trace.run_events() + ); +} diff --git a/crates/runvil-internal/engine/src/execute/tests/suite/prepare.rs b/crates/runvil-internal/engine/src/execute/tests/suite/prepare.rs index 8375722f3..7fe48b49b 100644 --- a/crates/runvil-internal/engine/src/execute/tests/suite/prepare.rs +++ b/crates/runvil-internal/engine/src/execute/tests/suite/prepare.rs @@ -1,17 +1,25 @@ -//! Prepare-pass tests that reach engine-only items: the per-run store -//! mount's claims isolation, and the test driver's prepare-then-run path -//! refusing an unsatisfiable prompt with the model-readable notice or -//! running a satisfiable one. The rest of the prepare suite runs against -//! the `runvil` facade. +//! Start tests that reach engine-only items: the per-run store mount's +//! claims isolation, the caller's role map filling the model set, and the +//! run's first step refusing an unsatisfiable prompt with the +//! model-readable notice or running a satisfiable one. The rest of the +//! suite runs against the `runvil` facade. +use std::collections::BTreeMap; use std::num::NonZeroU32; +use std::sync::{Arc, Mutex}; use crate::RunErrorKind; -use crate::execute::tests::context::test_context; +use crate::execute::context::ChainState; +use crate::execute::tests::context::test_config; +use crate::execute::{Effect, EffectAnswer, RunConfig}; +use crate::model::ModelBinding; +use crate::parser::Prompt; use crate::test_support::scenario::Scenario; use crate::test_support::scripted_chat::{ScriptedChat, text}; -use runvil_parser::test_support::parse_ok; -use runvil_types::models::{ModelId, ModelInfo, ThinkingMode}; +use runvil_model_client::client::{Completion, CompletionResult}; +use runvil_parser::test_support::{PromptSource, parse_ok}; +use runvil_types::event::Event; +use runvil_types::models::{ModelId, ModelInfo, ModelRole, ThinkingMode}; use runvil_vfs::Origin; /// A prompt declaring no Packs at all. @@ -28,15 +36,13 @@ const DECLARES_NOTHING: &str = concat!( #[test] fn two_runs_writing_the_same_store_path_do_not_conflict() { - let prompt = parse_ok(DECLARES_NOTHING); - let (ctx_a, _) = test_context("run-a").prepare(&prompt, None); - let (ctx_b, _) = test_context("run-b").prepare(&prompt, None); + let (ctx_a, ctx_b) = (test_config("run-a"), test_config("run-b")); let access_a = ctx_a - .vfs() + .vfs .acquire(Origin::new("run-a")) .expect("run a acquires"); let access_b = ctx_b - .vfs() + .vfs .acquire(Origin::new("run-b")) .expect("run b acquires"); let store_a = @@ -51,159 +57,420 @@ fn two_runs_writing_the_same_store_path_do_not_conflict() { assert_eq!(store_b.read("paper.md").expect("run b reads"), b"from b"); } -/// A prompt declaring one model role with a hard keyword and a context -/// minimum. -const DECLARES_ANALYST: &str = concat!( - "---\n", - "name: declares-analyst\n", - "description: d\n", - "runvil: 0\n", - "models:\n", - " analyst:\n", - " keywords: [frontier, thinking]\n", - " min_context: 200000\n", - " description: Deep analysis\n", - "---\n\n", - "# Title\n\n", - "## Only\n\n", - "```lua\n", - "return 'done'\n", - "```\n", -); - -/// Builds the one current model the caller chose, with the given context -/// window and thinking capability. -fn current_model(context: u32, thinking: ThinkingMode) -> ModelInfo { +/// The model `name` with the given context window and thinking capability. +fn model_named(name: &str, context: u32, thinking: ThinkingMode) -> ModelInfo { ModelInfo::new( - ModelId::gateway("current").expect("the id is valid"), - "The current model", + ModelId::gateway(name).expect("the id is valid"), + "A model for the prepare tests", NonZeroU32::new(context).expect("the context window is non-zero"), thinking, ) } +/// The one current model the caller chose, with the given context window +/// and thinking capability. +fn current_model(context: u32, thinking: ThinkingMode) -> ModelInfo { + model_named("current", context, thinking) +} + +/// The caller's role map holding `entries`. +fn role_map( + entries: impl IntoIterator, +) -> BTreeMap { + entries.into_iter().collect() +} + /// A scenario over `source` run under the one current model `model`. fn under_model(source: &str, name: &str, model: ModelInfo) -> Scenario { Scenario::new(source).name(name).current_model(model) } -#[test] -fn run_with_fixture_refuses_an_unsatisfiable_prompt_with_a_model_readable_notice() { - let model = current_model(32_000, ThinkingMode::Never); - let trace = under_model(DECLARES_ANALYST, "refuse", model).run(); +/// The aliases of the bindings in `bindings`, in order. +fn aliases(bindings: &[ModelBinding]) -> Vec<&str> { + bindings.iter().map(ModelBinding::alias).collect() +} + +/// A prompt declaring `front` and one section that returns `lua`. +fn declaring(front: &str, lua: &str) -> String { + PromptSource::new() + .front(front) + .section("Only", lua) + .build() +} + +/// The state a run of `prompt` builds over `config`, as `Run::new` builds +/// it. +fn state_of(prompt: &Prompt, config: RunConfig) -> ChainState { + ChainState::root(Arc::new(prompt.clone()), "", config) +} + +/// The refusal notice of `scenario`'s run, after asserting that its first +/// step failed it as `RequirementsUnmet` with no run events and no +/// effects. +#[track_caller] +fn refusal(scenario: Scenario) -> String { + let trace = scenario.run(); let error = trace.error(); - assert_eq!(error.kind(), RunErrorKind::RequirementsUnmet); - let notice = error.to_string(); - // The notice is written to be read by a model: it names the role, - // each failed check, and required versus actual. + assert_eq!(error.kind(), RunErrorKind::RequirementsUnmet, "{error}"); assert!( - notice.starts_with("the environment cannot satisfy this prompt:"), - "the notice opens with the standing refusal line: {notice}" + trace.run_events().is_empty(), + "a refused run reports no run events: {:?}", + trace.run_events() ); - assert!( - notice.contains( - "role 'analyst': requires a context of at least 200000 tokens; \ - the current model provides 32000" - ), - "the notice gives required versus actual context: {notice}" + assert!(trace.effects().is_empty(), "a refused run issues nothing"); + error.to_string() +} + +#[test] +fn a_one_model_map_gives_all_five_roles_that_model() { + let model = model_named("only", 32_000, ThinkingMode::Switchable); + let state = state_of( + &parse_ok(DECLARES_NOTHING), + test_config("one-model").with_models(role_map([(ModelRole::Default, model.clone())])), ); - assert!( - notice.contains( - "role 'analyst': requires 'thinking'; \ - the current model's thinking capability is Never" - ), - "the notice gives required versus actual keywords: {notice}" + assert_eq!(state.refusal(), None); + let set = state.model_set(); + let bindings = set.bindings(); + assert_eq!( + aliases(bindings), + ["default", "strong", "small", "fast", "creative"], + "five bindings in role order, whether or not the prompt declares them" ); + for binding in bindings { + assert_eq!(binding.id(), model.id(), "role {}", binding.alias()); + } } #[test] -fn run_with_fixture_prepares_implicitly_and_runs_a_satisfiable_prompt() { - // The scenario prepares implicitly, and the declared role's - // requirements are met by the current model. - let model = current_model(200_000, ThinkingMode::Always); - let trace = under_model(DECLARES_ANALYST, "implicit", model).run(); - assert_eq!(trace.text(), "done"); +fn a_map_with_a_strong_entry_gives_strong_that_model_and_the_other_roles_the_default() { + let default = model_named("default-model", 32_000, ThinkingMode::Switchable); + let strong = model_named("strong-model", 200_000, ThinkingMode::Always); + let state = state_of( + &parse_ok(DECLARES_NOTHING), + test_config("two-models").with_models(role_map([ + (ModelRole::Default, default.clone()), + (ModelRole::Strong, strong.clone()), + ])), + ); + assert_eq!(state.refusal(), None); + for binding in state.model_set().bindings() { + let expected = if binding.alias() == "strong" { + &strong + } else { + &default + }; + assert_eq!(binding.id(), expected.id(), "role {}", binding.alias()); + assert_eq!( + binding.context(), + expected.context(), + "role {}", + binding.alias() + ); + } } -/// A prompt declaring one `no-thinking` role whose section asks it one -/// question. -const DECLARES_NO_THINKING: &str = concat!( - "---\n", - "name: declares-no-thinking\n", - "description: d\n", - "runvil: 0\n", - "models:\n", - " writer:\n", - " keywords: [no-thinking]\n", - "---\n\n", - "# Title\n\n", - "## Only\n\n", - "```lua\n", - "models.use('writer')\n", - "return models.infer('ping')\n", - "```\n", -); +#[test] +fn a_context_minimum_above_the_filling_models_refuses_the_run_naming_that_role() { + let source = declaring("models: { strong: { min_context: 200000 } }", "return 'x'"); + let notice = refusal( + Scenario::new(source).current_model(current_model(32_000, ThinkingMode::Switchable)), + ); + assert_eq!( + notice, + "the environment cannot satisfy this prompt:\n\ + - role 'strong': requires a context of at least 200000 tokens; \ + the model filling it provides 32000" + ); +} #[test] -fn a_no_thinking_role_on_a_switchable_model_prepares_and_asks_for_thinking_off() { - let gateway = ScriptedChat::new(vec![text("pong")]); - let model = current_model(32_000, ThinkingMode::Switchable); - let trace = under_model(DECLARES_NO_THINKING, "switchable", model) +fn a_thinking_requirement_the_filling_model_fails_refuses_the_run_in_role_order() { + let source = declaring( + "models: { strong: { thinking: true }, small: { thinking: false } }", + "return 'x'", + ); + let never = model_named("never", 32_000, ThinkingMode::Never); + let always = model_named("always", 32_000, ThinkingMode::Always); + let notice = refusal( + Scenario::new(source) + .current_model(never) + .role_model(ModelRole::Small, always), + ); + assert_eq!( + notice, + "the environment cannot satisfy this prompt:\n\ + - role 'strong': requires 'thinking: true'; \ + the thinking capability of the model filling it is Never\n\ + - role 'small': requires 'thinking: false'; \ + the thinking capability of the model filling it is Always" + ); +} + +#[test] +fn a_role_binding_carries_the_description_of_the_model_that_fills_it() { + let prompt = parse_ok(&declaring("models: { strong: {} }", "return 'x'")); + let state = state_of( + &prompt, + test_config("description").with_models(role_map([( + ModelRole::Default, + current_model(32_000, ThinkingMode::Never), + )])), + ); + assert_eq!(state.refusal(), None); + let set = state.model_set(); + let strong = set.binding("strong").expect("the strong role is bound"); + assert_eq!(strong.description(), "A model for the prepare tests"); + assert_eq!(set.bindings().len(), 5); +} + +#[test] +fn thinking_true_on_strong_turns_thinking_on_for_strong_rounds_only() { + let source = declaring( + "models: { strong: { thinking: true } }", + " + models.use('strong') + local first = models.infer('one') + models.use('fast') + return first .. '|' .. models.infer('two')", + ); + let gateway = ScriptedChat::new(vec![text("a"), text("b")]); + let trace = Scenario::new(source) + .current_model(model_named("model-a", 32_000, ThinkingMode::Switchable)) .chat(&gateway) .run(); - // An unmet requirement would refuse the run, so the reply also shows - // that a switchable model can turn thinking off. - assert_eq!(trace.text(), "pong"); - let body = gateway - .last_request() - .expect("the round reaches the gateway"); - assert_eq!(body.options.model(), "current"); - assert_eq!(body.options.thinking(), Some(false)); + assert_eq!(trace.text(), "a|b"); + let thinking: Vec> = gateway + .requests() + .iter() + .map(|body| body.options.thinking()) + .collect(); + assert_eq!( + thinking, + [Some(true), None], + "the strong round asks for thinking and the fast round does not" + ); +} + +#[test] +fn a_non_empty_map_without_a_default_entry_leaves_the_roles_empty_and_refuses_the_run() { + let strong = model_named("strong-model", 200_000, ThinkingMode::Switchable); + let state = state_of( + &parse_ok(DECLARES_NOTHING), + test_config("unbound").with_models(role_map([(ModelRole::Strong, strong.clone())])), + ); + assert!( + state.model_set().bindings().is_empty(), + "the model set stays empty when the role map has no default" + ); + let notice = refusal(Scenario::new(DECLARES_NOTHING).role_model(ModelRole::Strong, strong)); + assert_eq!( + notice, + "the environment cannot satisfy this prompt:\n\ + - role 'default': requires a model; the caller's role map has no 'default' entry" + ); } #[test] -fn a_no_thinking_role_on_an_always_thinking_model_is_refused() { - let model = current_model(32_000, ThinkingMode::Always); - let trace = under_model(DECLARES_NO_THINKING, "always", model).run(); +fn an_empty_map_leaves_the_model_set_empty_and_a_model_call_raises_model_required() { + let state = state_of( + &parse_ok(DECLARES_NOTHING), + test_config("empty").with_models(BTreeMap::new()), + ); + assert_eq!(state.refusal(), None); + assert!(state.model_set().bindings().is_empty()); + + let asking = declaring("", "return models.infer('ping')"); + let trace = Scenario::new(asking).run(); let error = trace.error(); - assert_eq!(error.kind(), RunErrorKind::RequirementsUnmet); - let notice = error.to_string(); + assert_eq!(error.kind(), RunErrorKind::RequirementsUnmet, "{error}"); assert!( - notice.contains( - "role 'writer': requires 'no-thinking'; \ - the current model's thinking capability is Always" - ), - "the notice gives required versus actual keywords: {notice}" + error.to_string().contains("model binding required"), + "{error}" ); + + let silent = declaring("", "return 'no model needed'"); + assert_eq!(Scenario::new(silent).run().text(), "no model needed"); } -/// A prompt declaring one role that needs a long context. -const DECLARES_LONG_CONTEXT: &str = concat!( - "---\n", - "name: declares-long-context\n", - "description: d\n", - "runvil: 0\n", - "models:\n", - " reader:\n", - " min_context: 200000\n", - "---\n\n", - "# Title\n\n", - "## Only\n\n", - "Done.\n", -); +#[test] +fn a_prompt_with_no_models_block_runs_on_the_default_role_and_can_pick_another() { + let source = declaring( + "", + " + local first = models.infer('one') + models.use('strong') + return first .. '|' .. models.infer('two')", + ); + let gateway = ScriptedChat::new(vec![text("a"), text("b")]); + let trace = Scenario::new(source) + .current_model(model_named("only", 32_000, ThinkingMode::Switchable)) + .chat(&gateway) + .run(); + assert_eq!(trace.text(), "a|b"); + let served: Vec = gateway + .requests() + .iter() + .map(|body| body.options.model().to_owned()) + .collect(); + assert_eq!(served, ["only", "only"], "one model fills every role"); +} -/// An unmet requirement found at prepare refuses the run with the -/// model-readable notice text, line for line. #[test] -fn an_unmet_requirement_produces_todays_model_readable_notice() { - let model = current_model(32_000, ThinkingMode::Switchable); - let trace = under_model(DECLARES_LONG_CONTEXT, "notice", model).run(); - let error = trace.error(); - assert_eq!(error.kind(), RunErrorKind::RequirementsUnmet); +fn a_use_of_each_role_runs_its_round_on_the_model_that_fills_the_role() { + let source = declaring( + "", + " + models.use('strong') + local strong = models.infer('one') + models.use('fast') + return strong .. '|' .. models.infer('two')", + ); + let gateway = ScriptedChat::new(vec![text("a"), text("b")]); + let trace = Scenario::new(source) + .current_model(model_named("model-a", 32_000, ThinkingMode::Switchable)) + .role_model( + ModelRole::Strong, + model_named("model-b", 32_000, ThinkingMode::Switchable), + ) + .chat(&gateway) + .run(); + assert_eq!(trace.text(), "a|b"); + let served: Vec = gateway + .requests() + .iter() + .map(|body| body.options.model().to_owned()) + .collect(); assert_eq!( - error.to_string(), - "the environment cannot satisfy this prompt:\n\ - - role 'reader': requires a context of at least 200000 tokens; \ - the current model provides 32000" + served, + ["model-b", "model-a"], + "strong runs on B, and fast falls back to the default model A" + ); +} + +#[test] +fn a_strong_round_attributes_its_request_and_its_events_to_the_strong_model() { + let source = declaring( + "", + " + models.use('strong') + return models.infer('ping')", + ); + let requested = Arc::new(Mutex::new(Vec::::new())); + let recorder = Arc::clone(&requested); + let trace = Scenario::new(source) + .current_model(model_named("model-a", 32_000, ThinkingMode::Switchable)) + .role_model( + ModelRole::Strong, + model_named("model-b", 32_000, ThinkingMode::Switchable), + ) + .chat_with(move |effect| { + let Effect::Chat { binding, .. } = effect else { + panic!("a model round, got {effect:?}"); + }; + let requested_name = binding.completion_options().model().to_owned(); + recorder + .lock() + .expect("the recorder is not poisoned") + .push(requested_name.clone()); + // A caller labels the completion with the model the request named. + let completion = + Completion::from_result(CompletionResult::Text("pong".to_owned()), requested_name) + .expect("a text completion builds"); + EffectAnswer::Chat(Ok(Box::new(completion))) + }) + .run(); + assert_eq!(trace.text(), "pong"); + assert_eq!( + *requested.lock().expect("the recorder is not poisoned"), + ["model-b"], + "the request names the strong model" + ); + let reply_models: Vec<&str> = trace + .events() + .iter() + .filter_map(|event| match event { + Event::AssistantReply { model, .. } => Some(model.as_str()), + _ => None, + }) + .collect(); + assert_eq!( + reply_models, + ["model-b"], + "the round's reply event carries the strong model's name" + ); +} + +/// A scenario over `lua`, with `default` on model A and `strong` on model B. +fn two_role_scenario(lua: &str) -> Scenario { + Scenario::new(declaring("", lua)) + .current_model(model_named("model-a", 32_000, ThinkingMode::Switchable)) + .role_model( + ModelRole::Strong, + model_named("model-b", 32_000, ThinkingMode::Switchable), + ) +} + +#[test] +fn a_saved_handle_puts_a_later_round_back_on_its_model() { + let gateway = ScriptedChat::new(vec![text("a"), text("b")]); + let trace = two_role_scenario( + " + local prev = models.current() + models.use('strong') + local first = models.infer('one') + models.use(prev) + return first .. '|' .. models.infer('two')", + ) + .chat(&gateway) + .run(); + assert_eq!(trace.text(), "a|b"); + assert_eq!( + gateway.requested_models(), + ["model-b", "model-a"], + "the saved handle returns the section to the default model A" + ); +} + +#[test] +fn models_use_of_a_handle_applies_its_options_on_top_of_the_handles_model() { + let gateway = ScriptedChat::new(vec![text("a")]); + let trace = two_role_scenario( + " + local prev = models.current() + models.use(prev, { temperature = 0.2 }) + return models.infer('one')", + ) + .chat(&gateway) + .run(); + assert_eq!(trace.text(), "a"); + assert_eq!(gateway.requested_models(), ["model-a"]); + assert_eq!( + gateway.requests()[0] + .options + .temperature() + .map(crate::model::Temperature::get), + Some(0.2), + "the temperature applies on top of the handle's own binding" + ); +} + +#[test] +fn a_raw_id_handle_runs_the_next_round_on_its_gateway_model() { + let gateway = ScriptedChat::new(vec![text("a")]); + let trace = two_role_scenario( + " + models.use(models.get('vendor/raw-model')) + return models.infer('one')", + ) + .config(|ctx| ctx.with_ui(serde_json::json!({}))) + .chat(&gateway) + .run(); + assert_eq!(trace.text(), "a"); + assert_eq!( + gateway.requested_models(), + ["vendor/raw-model"], + "a raw-id handle from models.get is accepted by models.use" ); } +#[path = "prepare-refusals.rs"] +mod refusals; diff --git a/crates/runvil-internal/engine/src/execute/tests/suite/require/effects.rs b/crates/runvil-internal/engine/src/execute/tests/suite/require/effects.rs index f69b6ff38..8770a9435 100644 --- a/crates/runvil-internal/engine/src/execute/tests/suite/require/effects.rs +++ b/crates/runvil-internal/engine/src/execute/tests/suite/require/effects.rs @@ -17,16 +17,14 @@ fn effect_case(source: &PromptSource, chat: &ScriptedChat) -> Scenario { .bound() } -/// The fixture tools Pack and the `writer` role every effect case -/// declares. +/// The fixture tools Pack every effect case declares. fn effect_source() -> PromptSource { - PromptSource::new().front("uses: [tools]\nmodels: { writer: {} }") + PromptSource::new().front("uses: [tools]") } #[test] fn library_code_makes_store_model_and_tool_calls_while_it_loads() { let source = effect_source() - .h1("models.default('writer')") .section("Main", "return require '## loader'") .section( "loader", @@ -53,16 +51,16 @@ fn an_h1_library_sets_prompt_wide_state_every_later_section_sees() { require '## offers' messages:user('ping') models.loop() - return messages[#messages].content", + return var.tag .. ':' .. messages[#messages].content", ) - .section("setup", "models.default('writer')") + .section("setup", "var.tag = 'ready'") .section("offers", "tools.offer('tools/echo')"); let gateway = ScriptedChat::new(vec![text("pong")]); let trace = effect_case(&source, &gateway).run(); assert_eq!( trace.text(), - "pong", - "the loop runs under the H1 library's default" + "ready:pong", + "the loop runs after the H1 library set the state the section reads" ); let round = gateway .last_request() diff --git a/crates/runvil-internal/engine/src/execute/tests/suite/require/failures.rs b/crates/runvil-internal/engine/src/execute/tests/suite/require/failures.rs index 00635124f..2885ef258 100644 --- a/crates/runvil-internal/engine/src/execute/tests/suite/require/failures.rs +++ b/crates/runvil-internal/engine/src/execute/tests/suite/require/failures.rs @@ -16,7 +16,7 @@ use super::*; #[track_caller] fn run_with_preludes(source: &str, preludes: Vec) -> Trace { require_case(source.to_owned()) - .context(|ctx| ctx.with_preludes(preludes)) + .config(|ctx| ctx.with_preludes(preludes)) .run() } diff --git a/crates/runvil-internal/engine/src/execute/tests/suite/vfs.rs b/crates/runvil-internal/engine/src/execute/tests/suite/vfs.rs index d059d8bfd..df2eb85fd 100644 --- a/crates/runvil-internal/engine/src/execute/tests/suite/vfs.rs +++ b/crates/runvil-internal/engine/src/execute/tests/suite/vfs.rs @@ -1,7 +1,7 @@ //! The executor's `VfsRef` contract with its caller: an end-to-end run -//! over the prepared handle, and the seed-run-extract round trip a caller +//! over the caller's handle, and the seed-run-extract round trip a caller //! drives with no real files - seed the declared input through the run's -//! handle, prepare and run, extract the declared output, and charge a +//! handle, run, extract the declared output, and charge a //! missing output to the prompt's promise as an explicit contract error. //! Then the run-bounded scope: a run's `Done`, or its drop before `Done`, //! ends its scope however long the caller holds the store views it was @@ -9,9 +9,9 @@ use std::sync::Arc; -use crate::execute::RunResult; +use crate::execute::RunOutcome; use crate::execute::run::{Effect, IssuedEffect, Run, Step}; -use crate::execute::tests::context::{parse, test_context}; +use crate::execute::tests::context::{parse, test_config}; use crate::parser::Prompt; use crate::test_support::perform_locally; use crate::test_support::scenario::{EXECUTION, Scenario}; @@ -65,7 +65,7 @@ const MISSING_OUTPUT: &str = concat!( fn fresh_store_view(vfs: &VfsRef, origin: &str) -> Access { let access = vfs .acquire(Origin::new(origin)) - .expect("the prepared backend acquires"); + .expect("the caller's backend acquires"); runvil_vfs::detail::store_view(&access).expect("the handle declares a store") } @@ -103,9 +103,9 @@ fn seed_declared_input(vfs: &VfsRef, prompt: &Prompt, contents: &str) { } /// Seeds `source`'s declared input with `contents` and runs it over the -/// seeded handle through the prepare, which keeps the handle as given: -/// the test's seed-run-extract sequence is seed through the handle, -/// prepare and run, extract through the handle. Returns the run's text +/// seeded handle, which the run keeps as given: the test's +/// seed-run-extract sequence is seed through the handle, run, extract +/// through the handle. Returns the run's text /// and the handle. #[track_caller] fn seeded_run(source: &str, name: &str, contents: &str) -> (String, VfsRef) { @@ -119,7 +119,7 @@ fn seeded_run(source: &str, name: &str, contents: &str) -> (String, VfsRef) { fn an_end_to_end_run_threads_one_vfs_ref_through_every_section() { // The store survives the context-clearing section transition: one // section's write is the next section's read, over the run's - // prepared handle. + // handle. let source = prompt( "", &[ @@ -140,7 +140,7 @@ fn an_end_to_end_run_threads_one_vfs_ref_through_every_section() { } #[test] -fn a_caller_seeds_and_extracts_through_the_prepared_handle_with_no_real_files() { +fn a_caller_seeds_and_extracts_through_the_runs_handle_with_no_real_files() { let (text, vfs) = seeded_run(ROUND_TRIP_PROMPT, "vfs-round-trip", "the paper body"); assert_eq!(text, "done"); let store = fresh_store_view(&vfs, "round-trip extraction"); @@ -179,12 +179,11 @@ fn fanout_interleaving_is_invariant_across_memory_and_real_backends() { // identical. const FANOUT_STORE_WRITES: &str = include_str!("../../../../tests/prompts/execution/fanout-store-writes.md"); - // Both arms drive the raw caller-supplied handle contract (no prepare - // pass), so the caller's own backend serves the declared store in each. + // Both arms drive the raw caller-supplied handle contract, so the + // caller's own backend serves the declared store in each. let memory = Scenario::new(FANOUT_STORE_WRITES) .name("vfs-invariance-memory") .vfs(VfsRef::default()) - .unprepared() .run(); let temp = TempRoot::new("fanout_interleaving_is_invariant_across_memory_and_real_backends"); @@ -197,7 +196,6 @@ fn fanout_interleaving_is_invariant_across_memory_and_real_backends() { let real = Scenario::new(FANOUT_STORE_WRITES) .name("vfs-invariance-real") .vfs(real_vfs) - .unprepared() .run(); assert_eq!( @@ -230,7 +228,7 @@ const HELD_STORE: &str = "---\nname: held-store\ndescription: d\nrunvil: 0\n---\ /// Starts the `HELD_STORE` run over `vfs`. fn held_store_run(vfs: &VfsRef) -> Run { - let ctx = test_context(EXECUTION).with_vfs(vfs.clone()); + let ctx = test_config(EXECUTION).with_vfs(vfs.clone()); Run::new(Arc::new(parse(HELD_STORE)), "", ctx) } @@ -250,12 +248,12 @@ fn answer_holding(run: &mut Run, effects: Vec, held: &mut Vec (Run, RunResult, Vec) { +fn drive_holding_effects(vfs: &VfsRef) -> (Run, RunOutcome, Vec) { let mut run = held_store_run(vfs); let mut held = Vec::new(); loop { match run.step() { - Step::Done { result, .. } => return (run, result, held), + Step::Done { outcome, .. } => return (run, outcome, held), Step::Pending { effects, .. } => answer_holding(&mut run, effects, &mut held), } } @@ -265,10 +263,10 @@ fn drive_holding_effects(vfs: &VfsRef) -> (Run, RunResult, Vec) { fn a_run_ends_its_scope_at_done_while_the_caller_still_holds_its_store_views() -> Result<(), VfsError> { let vfs = VfsRef::default(); - let (run, result, held) = drive_holding_effects(&vfs); + let (run, outcome, held) = drive_holding_effects(&vfs); assert!( - matches!(&result, RunResult::Completed(text) if text == "from the run"), - "the run succeeds: {result:?}" + matches!(&outcome, RunOutcome::Completed { final_text: text } if text == "from the run"), + "the run succeeds: {outcome:?}" ); assert!( held.iter() diff --git a/crates/runvil-internal/engine/src/execute/tests/tasks-chain-end.rs b/crates/runvil-internal/engine/src/execute/tests/tasks-chain-end.rs index 713741e35..f080da2c8 100644 --- a/crates/runvil-internal/engine/src/execute/tests/tasks-chain-end.rs +++ b/crates/runvil-internal/engine/src/execute/tests/tasks-chain-end.rs @@ -48,7 +48,7 @@ fn a_chain_ending_with_live_author_tasks_fails_as_tasks_live_naming_the_ids() { CHILD_PARKED_ON_STORE_WRITE, ); let mut rig = Rig::run(scheduler_scenario(md)); - let error = rig.trace().error().inner(); + let error = rig.trace().error(); match error { Error::TasksLive { tasks } => { @@ -92,7 +92,7 @@ fn a_chain_failing_with_a_live_task_keeps_its_own_error_and_abandons_the_task() CHILD_PARKED_ON_STORE_WRITE, ); let mut rig = Rig::run(scheduler_scenario(md)); - let error = rig.trace().error().inner(); + let error = rig.trace().error(); assert!( !matches!(error, Error::TasksLive { .. }), @@ -160,7 +160,7 @@ fn a_returning_owner_abandons_its_running_task_as_owner_returned() { CHILD_PARKED_ON_HELD_ROUND, ); let rig = run_abandoning_a_running_task(scheduler_scenario(md), AbandonReason::OwnerReturned); - match rig.trace().error().inner() { + match rig.trace().error() { Error::TasksLive { tasks } => assert_eq!(tasks, &[task("0.0")]), other => panic!("expected tasks_live, got {other:?}"), } @@ -175,7 +175,7 @@ fn a_failing_owner_abandons_its_running_task_as_owner_failed() { CHILD_PARKED_ON_HELD_ROUND, ); let rig = run_abandoning_a_running_task(scheduler_scenario(md), AbandonReason::OwnerFailed); - let error = rig.trace().error().inner(); + let error = rig.trace().error(); assert!( error.to_string().contains("spawner boom"), "the owner keeps its own error over the leak: {error}" @@ -201,7 +201,7 @@ fn an_exhausted_tool_loop_abandons_its_running_task_as_tool_loop_exhausted() { .offer("echo", echo_tool()) .chat(&gateway); let rig = run_abandoning_a_running_task(scenario, AbandonReason::ToolLoopExhausted); - let error = rig.trace().error().inner(); + let error = rig.trace().error(); assert!( matches!(error, Error::ToolLoopExhausted), "the owner's own error stands: {error:?}" @@ -238,7 +238,7 @@ fn moving_spawner_prompt(movement: &str) -> String { fn assert_task_outlives_movement(md: String) { let mut rig = Rig::run(scheduler_scenario(md)); - match rig.trace().error().inner() { + match rig.trace().error() { Error::TasksLive { tasks } => assert_eq!( tasks, &[task("0.0")], @@ -341,7 +341,7 @@ fn a_task_spawned_in_h1_belongs_to_the_main_walk() { ```\n"; let mut rig = Rig::run(scheduler_scenario(md)); - match rig.trace().error().inner() { + match rig.trace().error() { Error::TasksLive { tasks } => assert_eq!( tasks, &[task("0.0")], diff --git a/crates/runvil-internal/engine/src/execute/tests/tasks.rs b/crates/runvil-internal/engine/src/execute/tests/tasks.rs index 5c9f1698d..d5130bb92 100644 --- a/crates/runvil-internal/engine/src/execute/tests/tasks.rs +++ b/crates/runvil-internal/engine/src/execute/tests/tasks.rs @@ -281,11 +281,10 @@ fn spawn_shares_calls_depth_cap() { // Either way the whole spawn cascade and the one refusal ran before // any answer arrived, which is what this test measures. let rig = Rig::run(scheduler_scenario(md)); - let outcome = rig.trace().result(); + let outcome = rig.trace().outcome(); assert!( - matches!(outcome, RunResult::Completed(_)) - || matches!(outcome, RunResult::Failure(error) - if matches!(error.inner(), Error::TasksLive { .. })), + matches!(outcome, RunOutcome::Completed { .. }) + || matches!(rig.trace().error(), Error::TasksLive { .. }), "unexpected outcome: {outcome:?}" ); diff --git a/crates/runvil-internal/engine/src/execute/tests/tool_call_access.rs b/crates/runvil-internal/engine/src/execute/tests/tool_call_access.rs index 5115076c9..304098168 100644 --- a/crates/runvil-internal/engine/src/execute/tests/tool_call_access.rs +++ b/crates/runvil-internal/engine/src/execute/tests/tool_call_access.rs @@ -268,15 +268,21 @@ fn a_cancelled_tasks_tool_call_is_joined_and_its_later_writes_are_refused() { let mut held: Option<(EffectId, Arc, EffectAnswer)> = None; let mut late = None; let result = loop { - let effects = match run.step() { - Step::Done { result, .. } => break result, - Step::Pending { effects, .. } => effects, + let (effects, withdrawn) = match run.step() { + Step::Done { outcome, .. } => break outcome, + Step::Pending { + effects, withdrawn, .. + } => (effects, withdrawn), }; if effects.is_empty() { - assert!(run.decided(), "only a decided run waits on nothing new"); let (id, access, answer) = held .take() .expect("the decided run waits on the held call's answer"); + assert_eq!( + withdrawn, + [id], + "only the step that decides the run waits on nothing new, and it withdraws the held call" + ); late = Some(access.write("late.txt", b"after the cancel")); run.resume(id, answer); continue; diff --git a/crates/runvil-internal/engine/src/execute/tests/tool_call_origins.rs b/crates/runvil-internal/engine/src/execute/tests/tool_call_origins.rs index f6797a50b..e800f2fce 100644 --- a/crates/runvil-internal/engine/src/execute/tests/tool_call_origins.rs +++ b/crates/runvil-internal/engine/src/execute/tests/tool_call_origins.rs @@ -56,15 +56,6 @@ fn script_call(section: &str, entry: &str) -> ToolCallOrigin { ToolCallOrigin::new(EXECUTION, section, ToolCaller::Script).with_entry(entry) } -/// The model each round asked for, in the order the rounds left. -fn request_models(gateway: &ScriptedChat) -> Vec { - gateway - .requests() - .iter() - .map(|body| body.options.model().to_owned()) - .collect() -} - /// A section that offers `echo` and an adhoc `grab` whose handler runs /// `handler`, then loops once and returns the final record's text. fn grab_then_loop(handler: &str) -> String { @@ -109,7 +100,7 @@ fn a_model_calls_origin_names_its_rounds_model_and_a_script_calls_names_none() { vec![ ToolCallOrigin::new(EXECUTION, "Only", ToolCaller::Model) .with_entry("0.1") - .with_model(loop_model_name("writer")), + .with_model(loop_model_name("default")), script_call("Only", "0.1"), ], "the model call names its round's model, and the script call after that round names none" @@ -122,7 +113,7 @@ fn a_section_that_switches_models_between_rounds_reports_each_rounds_model() { "tools.offer('tools/echo')\n\ messages:user('first')\n\ models.loop()\n\ - models.use('other')\n\ + models.use('strong')\n\ messages:user('second')\n\ models.loop()\n\ return messages[#messages].content", @@ -140,8 +131,8 @@ fn a_section_that_switches_models_between_rounds_reports_each_rounds_model() { assert_eq!( models(&rig), vec![ - Some(loop_model_name("writer")), - Some(loop_model_name("other")) + Some(loop_model_name("default")), + Some(loop_model_name("strong")) ], "each call names the model of the round that requested it" ); @@ -150,18 +141,18 @@ fn a_section_that_switches_models_between_rounds_reports_each_rounds_model() { #[test] fn an_adhoc_handlers_infer_on_another_model_leaves_the_batchs_later_call_on_its_rounds_model() { let (rig, gateway) = run_echo( - &grab_then_loop("return models.get('other'):infer('aside')"), + &grab_then_loop("return models.infer(models.get('strong'), 'aside')"), vec![grab_then_echo(), text("aside reply"), text("final answer")], ); assert_eq!(rig.trace().text(), "final answer"); assert_eq!( - request_models(&gateway), + gateway.requested_models(), ["test-model", "other-model", "test-model"], "the handler's round goes to another model between the loop's rounds" ); assert_eq!( models(&rig), - vec![Some(loop_model_name("writer"))], + vec![Some(loop_model_name("default"))], "the call after the handler names the batch round's model" ); } @@ -169,18 +160,18 @@ fn an_adhoc_handlers_infer_on_another_model_leaves_the_batchs_later_call_on_its_ #[test] fn an_adhoc_handlers_model_switch_leaves_the_batchs_later_call_on_its_rounds_model() { let (rig, gateway) = run_echo( - &grab_then_loop("models.use('other')\nreturn 'switched'"), + &grab_then_loop("models.use('strong')\nreturn 'switched'"), vec![grab_then_echo(), text("final answer")], ); assert_eq!(rig.trace().text(), "final answer"); assert_eq!( - request_models(&gateway), + gateway.requested_models(), ["test-model", "other-model"], "the switch steers the loop's next round" ); assert_eq!( models(&rig), - vec![Some(loop_model_name("writer"))], + vec![Some(loop_model_name("default"))], "the call after the handler names the batch round's model" ); } diff --git a/crates/runvil-internal/engine/src/execute/tests/tool_listings.rs b/crates/runvil-internal/engine/src/execute/tests/tool_listings.rs index de28e7845..ee4895e25 100644 --- a/crates/runvil-internal/engine/src/execute/tests/tool_listings.rs +++ b/crates/runvil-internal/engine/src/execute/tests/tool_listings.rs @@ -8,7 +8,7 @@ use runvil_parser::test_support::one_section; use runvil_types::event::Event; -use super::models_loop::catalog_scenario; +use super::models_loop::loop_scenario; use super::*; use crate::test_support::scenario::TEST_SEED; @@ -101,7 +101,7 @@ fn looped(section: &str, requested: &[&[(&str, &str, &str)]]) -> (Trace, Scripte let mut replies: Vec<_> = requested.iter().map(|batch| calls(batch)).collect(); replies.push(text("done")); let gateway = ScriptedChat::new(replies); - let trace = catalog_scenario(one_section(&format!( + let trace = loop_scenario(one_section(&format!( "{section}\n\ messages:user('list the tools')\n\ models.loop()\n\ @@ -119,7 +119,7 @@ fn looped(section: &str, requested: &[&[(&str, &str, &str)]]) -> (Trace, Scripte #[test] fn a_scripts_listing_call_returns_the_text_unwrapped_and_reports_it_untrusted() { - let trace = catalog_scenario(one_section("return tools.call('tools/listing', {})")) + let trace = loop_scenario(one_section("return tools.call('tools/listing', {})")) .tool(listing_tool()) .run(); assert_eq!( @@ -136,7 +136,7 @@ fn a_scripts_listing_call_returns_the_text_unwrapped_and_reports_it_untrusted() #[test] fn a_script_call_to_a_tool_without_the_flag_stays_wrapped() { - let trace = catalog_scenario(one_section( + let trace = loop_scenario(one_section( "return tools.call('tools/untrusted_echo', { value = 'hi' })", )) .tool(untrusted_echo_tool()) diff --git a/crates/runvil-internal/engine/src/execute/tests/tool_loop-scope-gate.rs b/crates/runvil-internal/engine/src/execute/tests/tool_loop-scope-gate.rs index 80e4e8835..1f185fa2b 100644 --- a/crates/runvil-internal/engine/src/execute/tests/tool_loop-scope-gate.rs +++ b/crates/runvil-internal/engine/src/execute/tests/tool_loop-scope-gate.rs @@ -40,7 +40,7 @@ fn model_calling_an_offered_but_unscoped_tool_is_a_hard_error() { "{\"value\":\"x\"}", )])]); let trace = scoped_loop("").chat(&gateway).run(); - let error = trace.error().inner(); + let error = trace.error(); match error { Error::OutOfScopeToolCall { name, @@ -78,7 +78,7 @@ fn model_calling_an_advertised_tool_by_canonical_id_gets_no_not_offered_suffix() "{\"value\":\"x\"}", )])]); let trace = scoped_loop("").chat(&gateway).run(); - let error = trace.error().inner(); + let error = trace.error(); match error { Error::OutOfScopeToolCall { name, @@ -114,7 +114,7 @@ fn model_calling_a_defined_but_unoffered_adhoc_tool_gets_no_not_offered_suffix() ) .chat(&gateway) .run(); - let error = trace.error().inner(); + let error = trace.error(); match error { Error::OutOfScopeToolCall { name, @@ -194,7 +194,7 @@ fn with_dynamic_offers_a_name_outside_both_sets_fails_listing_only_the_static_se // discovery tools because this section has them. let gateway = ScriptedChat::new(vec![calls(&[("call_1", "rogue", "{}")])]); let trace = scoped_loop(DYNAMIC_OFFERS).chat(&gateway).run(); - match trace.error().inner() { + match trace.error() { Error::OutOfScopeToolCall { name, global_exists, @@ -211,7 +211,7 @@ fn with_dynamic_offers_a_name_outside_both_sets_fails_listing_only_the_static_se } other => panic!("expected OutOfScopeToolCall, got {other:?}"), } - let message = trace.error().inner().to_string(); + let message = trace.error().to_string(); assert!( message.ends_with( "; a Dynamic Set tool is reached through GetDynamicTools and CallDynamicTool" @@ -227,7 +227,7 @@ fn a_section_without_a_dynamic_set_is_told_nothing_about_the_discovery_tools() { // so naming one would point a model at a tool its round cannot call. let gateway = ScriptedChat::new(vec![calls(&[("call_1", "rogue", "{}")])]); let trace = scoped_loop("").chat(&gateway).run(); - let message = trace.error().inner().to_string(); + let message = trace.error().to_string(); assert!( message.ends_with("aliases this section offers: [\"scoped\"]"), "the message ends at the Static Set: {message}" @@ -254,7 +254,7 @@ fn a_static_tool_wearing_a_discovery_wire_name_does_not_claim_a_dynamic_set() { .offer("CallDynamicTool", scoped_tool("call", "Call.").0) .chat(&gateway) .run(); - let error = trace.error().inner(); + let error = trace.error(); match error { Error::OutOfScopeToolCall { has_dynamic_set, @@ -290,7 +290,7 @@ fn model_calling_pure_unknown_tool_is_a_hard_error() { .offer("echo", echo_tool()) .chat(&gateway) .run(); - let error = trace.error().inner(); + let error = trace.error(); match error { Error::OutOfScopeToolCall { name, diff --git a/crates/runvil-internal/engine/src/execute/tests/tool_loop.rs b/crates/runvil-internal/engine/src/execute/tests/tool_loop.rs index d7214d9e6..56b04df93 100644 --- a/crates/runvil-internal/engine/src/execute/tests/tool_loop.rs +++ b/crates/runvil-internal/engine/src/execute/tests/tool_loop.rs @@ -46,7 +46,7 @@ fn tool_loop_gives_up_after_exactly_the_configured_cap() { .offer("echo", echo_tool()) .chat(&gateway) .run(); - let error = trace.error().inner(); + let error = trace.error(); assert!(matches!(error, Error::ToolLoopExhausted), "got {error:?}"); assert_eq!( gateway.call_count(), @@ -94,7 +94,7 @@ fn tool_loop_uses_the_default_cap_when_unspecified() { .offer("echo", echo_tool()) .chat(&gateway) .run(); - let error = trace.error().inner(); + let error = trace.error(); assert!(matches!(error, Error::ToolLoopExhausted), "got {error:?}"); assert_eq!(gateway.call_count(), default_cap); } @@ -219,7 +219,7 @@ fn repeated_calls_to_a_failing_tool_exit_at_the_iteration_cap() { .offer("echo", failing_tool()) .chat(&gateway) .run(); - let error = trace.error().inner(); + let error = trace.error(); assert!(matches!(error, Error::ToolLoopExhausted), "got {error:?}"); assert_eq!( gateway.call_count(), @@ -421,9 +421,9 @@ fn cancel_during_in_flight_tool_call_returns_promptly() { driving.cancel(); let trace = driving.finish(); assert!( - matches!(trace.result(), RunResult::Cancelled), + matches!(trace.outcome(), RunOutcome::Cancelled), "expected the cancelled run, got {:?}", - trace.result() + trace.outcome() ); assert!( trace diff --git a/crates/runvil-internal/engine/src/execute/tests/tool_scoping.rs b/crates/runvil-internal/engine/src/execute/tests/tool_scoping.rs index 959401b54..43ee240e1 100644 --- a/crates/runvil-internal/engine/src/execute/tests/tool_scoping.rs +++ b/crates/runvil-internal/engine/src/execute/tests/tool_scoping.rs @@ -113,7 +113,7 @@ fn a_section_offer_by_id_scopes_the_tool_and_dispatches_it() { let vm = SectionVm::new_for_section( &GuardNonce::from_seed(&TEST_SEED), &Arc::new(tools), - &Arc::new(Mutex::new(ModelSet::default())), + &Arc::new(ModelSet::default()), &null_emitter(), "Only", ) diff --git a/crates/runvil-internal/engine/src/execute/tests/unified_pipeline.rs b/crates/runvil-internal/engine/src/execute/tests/unified_pipeline.rs index 62e00f00d..8728c73ac 100644 --- a/crates/runvil-internal/engine/src/execute/tests/unified_pipeline.rs +++ b/crates/runvil-internal/engine/src/execute/tests/unified_pipeline.rs @@ -6,7 +6,6 @@ use runvil_parser::test_support::PromptSource; -use super::context::with_writer; use super::fixtures::echo_tool; use super::*; @@ -18,7 +17,7 @@ fn finite_pipeline_runs_the_unified_surface_end_to_end() { text("refined text"), ]); - let source = with_writer(PromptSource::new().front("uses: [tools]")) + let source = PromptSource::new().front("uses: [tools]") .prose( "Draft", " diff --git a/crates/runvil-internal/engine/src/execute/turn-tests.rs b/crates/runvil-internal/engine/src/execute/turn-tests.rs index 7b7fbd98a..9d14c7a80 100644 --- a/crates/runvil-internal/engine/src/execute/turn-tests.rs +++ b/crates/runvil-internal/engine/src/execute/turn-tests.rs @@ -5,7 +5,7 @@ //! events carry the round's id and the model the completion names. use runvil_model_client::client::RawExchange; -use runvil_types::emitter::{DebugMode, EventSink}; +use runvil_types::emitter::EventSink; use runvil_types::ids::RoundId; use runvil_types::metrics::{CallMetrics, Usage}; use serde_json::{Value, json}; @@ -50,12 +50,12 @@ fn usage_metrics(prompt_tokens: u32, completion_tokens: u32) -> CallMetrics { /// Reports `completion` as chat turn 3, round 5, and returns the events it /// pushed. fn report(completion: Completion) -> Vec { - report_with(completion, DebugMode::Off) + report_with(completion, false) } -/// Reports `completion` as chat turn 3, round 5, under `debug` and returns -/// the events it pushed. -fn report_with(completion: Completion, debug: DebugMode) -> Vec { +/// Reports `completion` as chat turn 3, round 5, capturing debug bodies +/// when `debug` is set, and returns the events it pushed. +fn report_with(completion: Completion, debug: bool) -> Vec { let sink = EventSink::default(); let emitter = Emitter::root(sink.clone(), "run-1", debug); let _ = report_model_turn( @@ -226,7 +226,7 @@ fn debug_capture_emits_the_request_and_response_from_the_raw_exchange() { let completion = canned() .with_raw(RawExchange::new(request.clone(), response.clone())) .with_finish_reason("stop"); - let events = report_with(completion, DebugMode::On); + let events = report_with(completion, true); assert_eq!(requests(&events), vec![request]); assert_eq!( responses(&events), @@ -252,7 +252,7 @@ fn debug_capture_emits_the_request_and_response_from_the_raw_exchange() { #[test] fn a_completion_with_no_raw_exchange_still_reports_its_pair_with_null_bodies() { - let events = report_with(canned(), DebugMode::On); + let events = report_with(canned(), true); assert_eq!(requests(&events), vec![Value::Null], "{events:?}"); assert_eq!( responses(&events), @@ -305,7 +305,7 @@ fn the_content_events_and_the_answer_record_name_the_model_the_completion_names( #[test] fn a_raw_exchange_is_not_captured_while_debug_is_off() { let completion = canned().with_raw(RawExchange::new(json!({}), json!({}))); - let events = report_with(completion, DebugMode::Off); + let events = report_with(completion, false); assert!(requests(&events).is_empty(), "{events:?}"); assert!(responses(&events).is_empty(), "{events:?}"); } diff --git a/crates/runvil-internal/engine/src/lib.rs b/crates/runvil-internal/engine/src/lib.rs index 49079b10c..5865469c8 100644 --- a/crates/runvil-internal/engine/src/lib.rs +++ b/crates/runvil-internal/engine/src/lib.rs @@ -32,9 +32,6 @@ use crate::error::{Error, Result}; // is re-exported here, so the `runvil` facade names // `runvil_engine::X`, not a module path. pub use crate::execute::{ - AnswerRecord, ChatAnswerRecord, Effect, EffectAnswer, EffectId, EffectRecord, FailureRecord, - IssuedEffect, MissingService, ModelBindings, PackToolSets, RequirementCheck, Requirements, - Round, Run, RunContext, RunError, RunErrorKind, RunLimits, RunResult, SourceLocation, Step, - ToolAnswerRecord, ToolCallOrigin, ToolCaller, UnavailablePack, UnmetRequirement, UsablePack, - perform_vfs_op, + Effect, EffectAnswer, EffectId, IssuedEffect, PackToolSets, Round, Run, RunConfig, + RunErrorKind, RunLimits, RunOutcome, Step, ToolCallOrigin, ToolCaller, perform_vfs_op, }; diff --git a/crates/runvil-internal/engine/src/lua.rs b/crates/runvil-internal/engine/src/lua.rs index 780b17d79..53a9b6a1c 100644 --- a/crates/runvil-internal/engine/src/lua.rs +++ b/crates/runvil-internal/engine/src/lua.rs @@ -25,7 +25,7 @@ pub(crate) use runvil_lua::{ UsageAnchor, VfsOp, VfsOutcome, YieldParse, current_catalog_packs, current_dynamic_set, current_static_set, exhaustion_detail, install_preludes, install_store_shims, install_ui, output_reserve, precheck, prepare_dispatch, prepare_model_dispatch, project_messages, - render_item, resolve_model_binding, wire_name_of, + render_item, wire_name_of, }; // The store operation behind `execute::perform_vfs_op`, the entry point diff --git a/crates/runvil-internal/engine/src/lua/tests.rs b/crates/runvil-internal/engine/src/lua/tests.rs index 9574768ec..5b5651649 100644 --- a/crates/runvil-internal/engine/src/lua/tests.rs +++ b/crates/runvil-internal/engine/src/lua/tests.rs @@ -113,18 +113,21 @@ fn set_up_vm(binds: &VmBinds<'_>) -> SectionVm { let vm = SectionVm::new_for_section( &GuardNonce::from_seed(&TEST_SEED), &Arc::new(binds.tools.clone()), - &Arc::new(Mutex::new(binds.models.clone())), + &Arc::new(binds.models.clone()), &emitter, binds.section, ) .expect("the section VM builds"); let sys = json!({}); let access = fresh_access(); + let set = binds.models; + let selection = Arc::new(Mutex::new(set.binding("default").cloned())); let setup = SectionVmSetup { args: "", argv: binds.argv, sys: &sys, access: &access, + selection: &selection, seed: binds.seed, emitter: &emitter, section_name: binds.section, diff --git a/crates/runvil-internal/engine/src/lua/tests/errors.rs b/crates/runvil-internal/engine/src/lua/tests/errors.rs index cd7f0618c..d90fef503 100644 --- a/crates/runvil-internal/engine/src/lua/tests/errors.rs +++ b/crates/runvil-internal/engine/src/lua/tests/errors.rs @@ -20,19 +20,30 @@ use crate::test_support::recording::null_emitter; use super::{compile_block, scheduler_vm, start, test_models}; #[test] -fn models_infer_rejects_a_second_argument() { - // `models.infer(prompt)` is the whole signature; a second argument - // (per-call options, or anything else) raises at the call site rather - // than being silently dropped. +fn models_infer_takes_a_prompt_or_a_model_and_a_prompt() { + // `models.infer(prompt)` and `models.infer(model, prompt)` are the + // whole signature; a third argument, or a second with no leading + // handle, raises at the call site rather than being silently dropped. let vm = scheduler_vm(&test_models(), None); let program = compile_block( - r#"local ok, err = pcall(models.infer, "yo", { temperature = 0 }) - assert(not ok, "a second argument must fail") - assert(tostring(err) == "models.infer takes (prompt)", tostring(err)) - return "rejected""#, + r#"local h = models.get("fast") + local out = {} + for _, call in ipairs({ + function() return models.infer("yo", { temperature = 0 }) end, + function() return models.infer(h, "yo", { temperature = 0 }) end, + }) do + local ok, err = pcall(call) + assert(not ok, "an argument past the acceptance shape must fail") + out[#out + 1] = type(err) .. "|" .. tostring(err.kind) .. "|" .. tostring(err) + end + return table.concat(out, "\n")"#, ); + let message = "models.infer takes (prompt) or (model, prompt)"; match vm.start_block_coro(&program).expect("the block runs") { - CoroStep::Done(LuaBlockResult::Returned(Some(text))) => assert_eq!(text, "rejected"), + CoroStep::Done(LuaBlockResult::Returned(Some(text))) => { + let line = format!("table|lua|{message}"); + assert_eq!(text, [line.clone(), line].join("\n")); + } other => panic!("expected the rejection return, got {other:?}"), } } @@ -40,67 +51,45 @@ fn models_infer_rejects_a_second_argument() { #[test] fn a_shim_argument_error_is_a_table_whose_tostring_is_the_message() { // Every failure that reaches Lua is a `{ kind, message, ... }` table: - // `tostring` (and `..`) gives exactly the message an author saw before, - // and a caller that branches reads `kind`. A shim's own argument error - // is an authoring error, so its kind is `lua`. + // `tostring` (and `..`) gives exactly the message, and a caller that + // branches reads `kind`. A shim's own argument error is an authoring + // error, so its kind is `lua`. let vm = scheduler_vm(&test_models(), None); let program = compile_block( r#"local h = models.get("fast") - local ok, err = pcall(h.infer, h, "yo", { temperature = 0 }) + local ok, err = pcall(models.infer, h, "yo", { temperature = 0 }) assert(not ok, "an argument past the prompt must fail") return type(err) .. "|" .. tostring(err.kind) .. "|" .. err"#, ); match vm.start_block_coro(&program).expect("the block runs") { CoroStep::Done(LuaBlockResult::Returned(Some(text))) => { - assert_eq!(text, "table|lua|handle:infer takes (prompt)"); + assert_eq!( + text, + "table|lua|models.infer takes (prompt) or (model, prompt)" + ); } other => panic!("expected the rejection return, got {other:?}"), } } #[test] -fn a_handle_passed_to_models_infer_is_the_pointed_error() { - // The handle-first form is gone; the error names the method that - // replaced it, before any request leaves. - let vm = scheduler_vm(&test_models(), None); - let program = compile_block( - r#"local ok, err = pcall(models.infer, models.get("fast"), "yo") - assert(not ok, "a handle first must fail") - return tostring(err.kind) .. "|" .. tostring(err)"#, - ); - match vm.start_block_coro(&program).expect("the block runs") { - CoroStep::Done(LuaBlockResult::Returned(Some(text))) => assert_eq!( - text, - "lua|models.infer takes (prompt); call handle:infer(prompt) to run on a model handle" - ), - other => panic!("expected the rejection return, got {other:?}"), - } -} - -#[test] -fn a_handles_infer_called_without_its_receiver_is_the_colon_error() { - // A dot call passes the prompt where the receiver belongs, and the - // function taken off the handle and called bare has no receiver. +fn a_handle_carries_no_call_entry_of_its_own() { + // A handle is a pure inspectable value: `infer` and `loop` are names + // it does not have, and the namespace functions are the only way to run + // a model. let vm = scheduler_vm(&test_models(), None); let program = compile_block( r#"local h = models.get("fast") - local run = h.infer local out = {} - for _, call in ipairs({ - function() return h.infer("yo") end, - function() return run("yo") end, - function() return h.infer() end, - }) do - local ok, err = pcall(call) - assert(not ok, "a call without the receiver must fail") - out[#out + 1] = tostring(err.kind) .. "|" .. tostring(err) + for _, name in ipairs({ "infer", "loop" }) do + local ok = pcall(function() return h[name] end) + out[#out + 1] = tostring(ok) end - return table.concat(out, "\n")"#, + return type(h) .. "|" .. table.concat(out, "|")"#, ); - let colon = "lua|call infer on a model handle with a colon: handle:infer(prompt)"; match vm.start_block_coro(&program).expect("the block runs") { CoroStep::Done(LuaBlockResult::Returned(Some(text))) => { - assert_eq!(text, [colon; 3].join("\n")); + assert_eq!(text, "userdata|false|false"); } other => panic!("expected the rejection return, got {other:?}"), } @@ -151,7 +140,7 @@ fn an_engine_callback_failure_caught_by_pcall_is_the_same_error_table() { let vm = scheduler_vm(&test_models(), None); let program = compile_block( r#"local ok, err = pcall(models.get, "missing") - assert(not ok, "an unbound alias must fail") + assert(not ok, "a label outside the five roles must fail") local first = type(err) .. "|" .. tostring(err.kind) .. "|" .. tostring(err) local ok2, err2 = pcall(function() return sys.nothing end) assert(not ok2, "an unknown sys field must fail") @@ -166,7 +155,7 @@ fn an_engine_callback_failure_caught_by_pcall_is_the_same_error_table() { CoroStep::Done(LuaBlockResult::Returned(Some(text))) => { assert_eq!( text, - "table|lua|models.get alias \"missing\" is not a bound model role\n\ + "table|lua|models.get label \"missing\" is not a model role: expected default, strong, small, fast or creative\n\ table|lua|runtime error: unknown sys field 'nothing'\n\ table|lua" ); @@ -222,7 +211,7 @@ fn an_uncaught_lua_kind_shim_raise_keeps_the_mapped_runtime_error() { // 40 and fails at chunk line 2, so the mapped author frame is line 41. let vm = scheduler_vm(&test_models(), None); let program = LuaProgram::compile( - "local h = models.get(\"fast\")\nh:infer(\"a\", \"b\")", + "local h = models.get(\"fast\")\nmodels.infer(h, \"a\", \"b\")", "section `Test` prologue", NonZeroU32::new(40).expect("40 is non-zero"), &null_emitter(), @@ -232,7 +221,7 @@ fn an_uncaught_lua_kind_shim_raise_keeps_the_mapped_runtime_error() { match vm.start_block_coro(&program) { Err(Error::LuaRuntime { message, .. }) => { assert!( - message.contains("handle:infer takes (prompt)"), + message.contains("models.infer takes (prompt) or (model, prompt)"), "the mapped error keeps the shim's message: {message}" ); assert!( diff --git a/crates/runvil-internal/engine/src/lua/tests/globals.rs b/crates/runvil-internal/engine/src/lua/tests/globals.rs index e88d7a4ee..e60728e1b 100644 --- a/crates/runvil-internal/engine/src/lua/tests/globals.rs +++ b/crates/runvil-internal/engine/src/lua/tests/globals.rs @@ -24,9 +24,8 @@ use crate::lua::{Argv, SectionVm, ToolSet}; use crate::model::ModelSet; use crate::test_support::scenario::test_binding; -/// The role labels every fully bound VM fills: one plain label, and one -/// that names an Engine global. -const ROLE_LABELS: [&str; 2] = ["writer", "tools"]; +/// The role labels every fully bound VM fills. +const ROLE_LABELS: [&str; 2] = ["default", "strong"]; /// One role bound under each of [`ROLE_LABELS`]. fn roles() -> ModelSet { @@ -147,15 +146,15 @@ fn no_role_label_is_a_global_after_section_or_h1_setup() { ("writable", Argv::Writable(Some(&argv))), ] { let vm = fully_bound_vm(argv); - let (writer, tools, call): (String, String, String) = vm + let (default, strong, call): (String, String, String) = vm .lua() - .load("return type(writer), type(tools), type(tools.call)") + .load("return type(default), type(strong), type(tools.call)") .eval() .expect("the globals probe evaluates"); assert_eq!( - (writer.as_str(), tools.as_str(), call.as_str()), - ("nil", "table", "function"), - "a role label installs no global and leaves a same-named Engine global in place \ + (default.as_str(), strong.as_str(), call.as_str()), + ("nil", "nil", "function"), + "a role label installs no global and leaves the Engine globals in place \ (argv {mode})" ); } diff --git a/crates/runvil-internal/engine/src/lua/tests/shims.rs b/crates/runvil-internal/engine/src/lua/tests/shims.rs index 7bc677445..285343e36 100644 --- a/crates/runvil-internal/engine/src/lua/tests/shims.rs +++ b/crates/runvil-internal/engine/src/lua/tests/shims.rs @@ -331,7 +331,7 @@ fn tools_call_rejects_a_first_argument_that_is_neither_a_string_nor_a_tool() { } #[test] -fn a_handles_infer_runs_on_its_frozen_binding() { +fn a_leading_handle_infer_runs_on_its_frozen_binding() { let vm = scheduler_vm(&test_models(), None); let request = yielded_request( &vm, @@ -340,7 +340,7 @@ fn a_handles_infer_runs_on_its_frozen_binding() { local u = models.use("fast") assert(h.name == "fast" and h.model_id == "test-model") assert(u.name == "fast") - return h:infer("yo") + return models.infer(h, "yo") "#, ); match request { @@ -359,7 +359,7 @@ fn a_handles_infer_runs_on_its_frozen_binding() { #[test] fn a_handle_from_models_get_infers_without_a_proxy() { let vm = scheduler_vm(&test_models(), None); - match yielded_request(&vm, r#"return models.get("fast"):infer("yo")"#) { + match yielded_request(&vm, r#"return models.infer(models.get("fast"), "yo")"#) { Request::Infer { prompt, binding: Some(binding), @@ -372,25 +372,32 @@ fn a_handle_from_models_get_infers_without_a_proxy() { } #[test] -fn model_handles_carry_infer_and_loop_methods() { - // A model handle stays a frozen, inspectable userdata whose `infer` - // and `loop` fields read the shim's methods; a field the handle does - // not have still fails to read. +fn model_handles_expose_their_fields_and_no_methods() { + // A model handle is a frozen, inspectable userdata: every documented + // field reads, and a callable name it does not have raises. let vm = scheduler_vm(&test_models(), None); - let (is_userdata, methods, read_failed): (bool, String, bool) = vm + let (is_userdata, fields, read_failed): (bool, String, bool) = vm .lua() .load( r#" local h = models.get("fast") local ok = pcall(function() return h.nothing end) + local inferred = pcall(function() return h.infer end) + local looped = pcall(function() return h.loop end) return type(h) == "userdata", - type(h.infer) .. "|" .. type(h.loop), - not ok + type(h.name) .. "|" .. type(h.label) .. "|" .. type(h.model_id) + .. "|" .. type(h.description) .. "|" .. type(h.context) + .. "|" .. type(h.provider) .. "|" .. tostring(h.thinking) + .. "|" .. tostring(h.temperature) .. "|" .. tostring(h.max_tokens), + not ok and not inferred and not looped "#, ) .eval() .expect("the handle probe evaluates"); assert!(is_userdata, "handles install as bare userdata"); - assert_eq!(methods, "function|function"); - assert!(read_failed, "a handle has no `nothing` field to read"); + assert_eq!(fields, "string|string|string|string|number|nil|nil|nil|nil"); + assert!( + read_failed, + "a handle has no `nothing`, `infer`, or `loop` field to read" + ); } diff --git a/crates/runvil-internal/engine/src/model.rs b/crates/runvil-internal/engine/src/model.rs index 3e770be98..252de57ea 100644 --- a/crates/runvil-internal/engine/src/model.rs +++ b/crates/runvil-internal/engine/src/model.rs @@ -2,7 +2,7 @@ //! includes ([`Message`], [`ToolSchema`], [`ModelBinding`]) and what its //! answer returns ([`Completion`], [`CompletionResult`], //! [`CompletionError`]), plus the prompt-local binding vocabulary -//! ([`ModelSet`], [`ModelView`], [`ModelInvocation`]) and the model +//! ([`ModelSet`], [`ModelInvocation`]) and the model //! identity a run's roles are bound to ([`ModelInfo`], [`ModelId`]). //! //! This module is the executor's import surface for the private @@ -30,10 +30,9 @@ pub(crate) use runvil_model_client::client::{ // documented. pub(crate) use runvil_model_client::detail; pub(crate) use runvil_model_client::model::{ - CompletionError, CompletionErrorKind, ModelBinding, ModelInvocation, ModelSet, ModelView, - Temperature, + CompletionError, CompletionErrorKind, ModelBinding, ModelInvocation, ModelSet, Temperature, }; -pub(crate) use runvil_types::models::{ModelId, ModelInfo, ThinkingMode}; +pub(crate) use runvil_types::models::{ModelId, ModelInfo, ModelRole, ThinkingMode}; #[cfg(test)] mod tests; diff --git a/crates/runvil-internal/engine/src/model/tests-always.rs b/crates/runvil-internal/engine/src/model/tests-always.rs index cb3c303da..942fa4ecd 100644 --- a/crates/runvil-internal/engine/src/model/tests-always.rs +++ b/crates/runvil-internal/engine/src/model/tests-always.rs @@ -1,154 +1,63 @@ -//! `models.default` (label form) integration tests. +//! The `default` role as the baseline a section starts on. use super::*; use runvil_model_client::model::CompletionOptions; #[test] -fn models_default_takes_a_label_and_parks_the_prompt_wide_default() { +fn a_section_starts_on_the_default_role_without_use() { let models = shared_models(vec![bound_role( - "writer", + "default", "A tiny model", "small", 8_192, Some(false), - &["no-thinking"], )]); let vm = section_vm_with_models(&models, &null_emitter(), "Section").expect("the section VM builds"); - vm.run_chunk( - &chunk(r#"models.default("writer")"#), - &null_emitter(), - "Section", - ) - .expect("a bound label becomes the default"); - assert_eq!( - models.lock().expect("set lock").default_alias(), - Some("writer") - ); - vm.teardown(&null_emitter(), "Section"); -} - -#[test] -fn models_default_returns_an_inspectable_handle() { - let models = shared_models(vec![bound_role( - "writer", - "A tiny model", - "small", - 8_192, - Some(false), - &["no-thinking", "fast"], - )]); - let vm = - section_vm_with_models(&models, &null_emitter(), "Section").expect("the section VM builds"); - vm.run_chunk( - &chunk( - r#"local model = models.default("writer") - assert(model.label == "writer") - assert(model.name == "writer") - assert(model.model_id == "small") - assert(model.description == "A tiny model") - assert(model.context == 8192) - assert(model.thinking == false) - assert(#model.capabilities == 2) - assert(model.capabilities[1] == "no-thinking") - assert(model.capabilities[2] == "fast")"#, - ), - &null_emitter(), - "Section", - ) - .expect("the handle exposes the role label and the full keyword set"); - vm.teardown(&null_emitter(), "Section"); -} - -#[test] -fn models_default_rejects_an_unbound_label() { - let models = shared_models(vec![bound_role( - "writer", - "A tiny model", - "small", - 8_192, - None, - &[], - )]); - let vm = - section_vm_with_models(&models, &null_emitter(), "Section").expect("the section VM builds"); - let error = vm - .run_chunk( - &chunk(r#"models.default("ghost")"#), - &null_emitter(), - "Section", - ) - .expect_err("an unbound label is a hard error"); - assert!( - error - .to_string() - .contains("models.default label \"ghost\" is not a bound model role"), - "the rejection names the label: {error}" - ); + let model = resolve_section_model(&vm).expect("the resolution reads the shared set"); + assert_eq!(model.as_ref().map(ModelBinding::alias), Some("default")); + let opts = model.as_ref().map(ModelBinding::completion_options); + let expected = CompletionOptions::new("small").with_thinking(false); + assert_eq!(opts, Some(expected)); vm.teardown(&null_emitter(), "Section"); } #[test] -fn models_default_is_idempotent_and_never_changes_mid_run() { +fn models_get_returns_an_inspectable_handle_without_selecting() { let models = shared_models(vec![ - bound_role("writer", "A tiny model", "small", 8_192, None, &[]), + bound_role("default", "A tiny model", "small", 8_192, Some(false)), bound_role( - "critic", + "strong", "A careful analysis model", "analyst", 131_072, None, - &[], ), ]); let vm = section_vm_with_models(&models, &null_emitter(), "Section").expect("the section VM builds"); - // A library every section requires re-runs in each, so re-naming the - // same default is a no-op. vm.run_chunk( - &chunk(r#"models.default("writer"); models.default("writer")"#), + &chunk( + r#"local model = models.get("default") + assert(model.label == "default") + assert(model.name == "default") + assert(model.model_id == "small") + assert(model.description == "A tiny model") + assert(model.context == 8192) + assert(model.thinking == false) + models.get("strong")"#, + ), &null_emitter(), "Section", ) - .expect("re-naming the same default is a no-op"); - let error = vm - .run_chunk( - &chunk(r#"models.default("critic")"#), - &null_emitter(), - "Section", - ) - .expect_err("the prompt-wide default cannot change mid-run"); - assert!( - error - .to_string() - .contains("models.default is already \"writer\""), - "the refusal names the parked default: {error}" + .expect("the handle exposes the role label and its fields"); + let current = resolve_section_model(&vm) + .expect("the resolution reads the shared set") + .expect("the section has a current model"); + assert_eq!( + current.alias(), + "default", + "models.get inspects a role without selecting it" ); vm.teardown(&null_emitter(), "Section"); } - -#[test] -fn models_default_resolves_the_section_model_without_use() { - let models = shared_models(vec![bound_role( - "writer", - "A tiny model", - "small", - 8_192, - Some(false), - &["no-thinking"], - )]); - let vm = - section_vm_with_models(&models, &null_emitter(), "Section").expect("the section VM builds"); - vm.run_chunk( - &chunk(r#"models.default("writer")"#), - &null_emitter(), - "Section", - ) - .expect("the default parks"); - let model = resolve_section_model(&vm).expect("the resolution reads the shared set"); - assert_eq!(model.as_ref().map(ModelBinding::alias), Some("writer")); - let opts = model.as_ref().map(ModelBinding::completion_options); - let expected = CompletionOptions::new("small").with_thinking(false); - assert_eq!(opts, Some(expected)); - vm.teardown(&null_emitter(), "Section"); -} diff --git a/crates/runvil-internal/engine/src/model/tests-integration.rs b/crates/runvil-internal/engine/src/model/tests-integration.rs index 697943bd8..d84034f38 100644 --- a/crates/runvil-internal/engine/src/model/tests-integration.rs +++ b/crates/runvil-internal/engine/src/model/tests-integration.rs @@ -3,44 +3,35 @@ use super::*; #[test] -fn models_use_selects_a_bound_role_by_label() { - let models = shared_models(vec![bound_role( - "analyst", - "A careful analysis model", - "analyst", - 131_072, - Some(true), - &["thinking", "frontier"], - )]); +fn models_use_selects_a_role_by_label() { + let models = shared_models(vec![ + bound_role("default", "A tiny model", "small", 8_192, None), + bound_role( + "strong", + "A careful analysis model", + "analyst", + 131_072, + Some(true), + ), + ]); let vm = section_vm_with_models(&models, &null_emitter(), "Section").expect("the section VM builds"); vm.run_chunk( - &chunk(r#"models.use("analyst")"#), + &chunk(r#"models.use("strong")"#), &null_emitter(), "Section", ) - .expect("a bound label selects"); + .expect("a role label selects"); let model = resolve_section_model(&vm).expect("the resolution reads the selection"); let model = model.expect("a selection resolves"); - assert_eq!(model.alias(), "analyst"); + assert_eq!(model.alias(), "strong"); assert_eq!(model.invocation().thinking, Some(true)); - assert_eq!( - model.capabilities(), - &["thinking".to_owned(), "frontier".to_owned()] - ); vm.teardown(&null_emitter(), "Section"); } #[test] -fn no_use_or_default_leaves_the_section_unbound() { - let models = shared_models(vec![bound_role( - "analyst", - "A careful analysis model", - "analyst", - 131_072, - None, - &[], - )]); +fn a_section_with_no_models_has_no_current_model() { + let models = shared_models(Vec::new()); let vm = section_vm_with_models(&models, &null_emitter(), "Section").expect("the section VM builds"); let model = resolve_section_model(&vm).expect("the resolution reads the shared set"); @@ -49,28 +40,49 @@ fn no_use_or_default_leaves_the_section_unbound() { } #[test] -fn models_use_rejects_an_unbound_label() { +fn models_use_rejects_a_label_outside_the_five_roles() { let models = shared_models(vec![bound_role( - "analyst", - "A careful analysis model", - "analyst", - 131_072, + "default", + "A tiny model", + "small", + 8_192, None, - &[], )]); let vm = section_vm_with_models(&models, &null_emitter(), "Section").expect("the section VM builds"); let error = vm .run_chunk( - &chunk(r#"models.use("missing")"#), + &chunk(r#"models.use("reviewer")"#), + &null_emitter(), + "Section", + ) + .expect_err("a label outside the five roles must fail"); + let rendered = error.to_string(); + assert!( + rendered.contains( + "models.use label \"reviewer\" is not a model role: expected default, strong, small, fast or creative" + ), + "the error must name the label and the five roles: {rendered}" + ); + vm.teardown(&null_emitter(), "Section"); +} + +#[test] +fn models_use_with_no_model_for_the_role_says_the_caller_supplied_none() { + let models = shared_models(Vec::new()); + let vm = + section_vm_with_models(&models, &null_emitter(), "Section").expect("the section VM builds"); + let error = vm + .run_chunk( + &chunk(r#"models.use("default")"#), &null_emitter(), "Section", ) - .expect_err("an unbound label must fail"); + .expect_err("a role with no model must fail"); let rendered = error.to_string(); assert!( - rendered.contains("models.use label \"missing\" is not a bound model role"), - "the error must name the unbound label: {rendered}" + rendered.contains("the caller supplied no model"), + "the error must say the caller supplied no model: {rendered}" ); vm.teardown(&null_emitter(), "Section"); } diff --git a/crates/runvil-internal/engine/src/model/tests.rs b/crates/runvil-internal/engine/src/model/tests.rs index 8fbf36c9a..d4f43b9f2 100644 --- a/crates/runvil-internal/engine/src/model/tests.rs +++ b/crates/runvil-internal/engine/src/model/tests.rs @@ -1,10 +1,10 @@ -//! Tests for resolving a section's model binding through the VM and shared model set. +//! Tests for reading a section's current model through the VM and the shared model set. use std::num::NonZeroU32; -use std::sync::{Arc, Mutex}; +use std::sync::Arc; use super::*; -use crate::lua::{SectionVm, ToolSet, resolve_model_binding}; +use crate::lua::{SectionVm, ToolSet}; use crate::test_support::fresh_access; use crate::test_support::recording::null_emitter; use crate::test_support::scenario::TEST_SEED; @@ -35,16 +35,14 @@ fn gateway_id(name: &str) -> ModelId { ModelId::gateway(name).expect("test model alias is valid") } -/// A bound role as prepare's fill records it: label, description, resolved -/// identity, the hard-keyword thinking switch as the frozen invocation, and -/// the role's keyword set. +/// A bound role as the run's fill records it: label, description, resolved +/// identity, and the thinking switch as the frozen invocation. fn bound_role( label: &str, description: &str, model: &str, window: u32, thinking: Option, - capabilities: &[&str], ) -> ModelBinding { ModelBinding::new( label, @@ -53,16 +51,15 @@ fn bound_role( ModelInvocation::new().with_thinking(thinking), ctx(window), ) - .with_capabilities(capabilities.iter().map(|word| (*word).to_owned()).collect()) } /// Shares a model set the way the run shares its own. -fn shared_models(bindings: Vec) -> Arc> { +fn shared_models(bindings: Vec) -> Arc { let mut set = ModelSet::default(); for binding in bindings { set.push(binding); } - Arc::new(Mutex::new(set)) + Arc::new(set) } /// Shares an empty tool set. @@ -72,7 +69,7 @@ fn shared_tools() -> Arc { /// A section VM over `models` with empty Engine values injected. fn section_vm_with_models( - models: &Arc>, + models: &Arc, emitter: &Emitter, section: &str, ) -> Result { @@ -87,11 +84,9 @@ fn section_vm_with_models( .map_err(|failure| Error::from(failure.teardown(emitter, section))) } -/// Reads the section's effective model binding through a view over the VM's -/// shared set, mirroring the Engine's read path. +/// Reads the section's current model the way the Engine's read path does. fn resolve_section_model(vm: &SectionVm) -> Result> { - let (models, runtime) = vm.model_bag_handles()?; - resolve_model_binding(&Mutex::new(models), &runtime).map_err(Error::from) + vm.current_model().map_err(Error::from) } #[path = "tests-always.rs"] diff --git a/crates/runvil-internal/engine/src/parser.rs b/crates/runvil-internal/engine/src/parser.rs index e989102ce..ad565ed72 100644 --- a/crates/runvil-internal/engine/src/parser.rs +++ b/crates/runvil-internal/engine/src/parser.rs @@ -19,5 +19,5 @@ pub(crate) use runvil_parser::detail; pub(crate) use runvil_parser::{ - Block, BodyText, ModelKeyword, ParseError, ParseErrorKind, Prompt, Section, + Block, BodyText, ParseError, ParseErrorKind, Prompt, RoleRequirements, Section, }; diff --git a/crates/runvil-internal/engine/src/test_support.rs b/crates/runvil-internal/engine/src/test_support.rs index 9a34a5571..e30c1f67d 100644 --- a/crates/runvil-internal/engine/src/test_support.rs +++ b/crates/runvil-internal/engine/src/test_support.rs @@ -30,9 +30,7 @@ use runvil_types::event::Event; use runvil_types::tools::ToolError; use crate::Error; -use crate::execute::{ - Effect, EffectAnswer, EffectId, IssuedEffect, Run, RunError, RunResult, Step, -}; +use crate::execute::{Effect, EffectAnswer, EffectId, IssuedEffect, Run, RunOutcome, Step}; use crate::lua::run_store_op; #[cfg(test)] @@ -99,15 +97,14 @@ pub fn perform_locally( } /// Drives `run` to its end on the calling thread, performing every effect -/// through `perform` as it is issued, and returns the run's result with +/// through `perform` as it is issued, and returns the run's outcome with /// every event it reported, in order. /// /// The driver is the simplest correct caller of a run. After each `step` /// it answers the step's effects in issue order, each through `perform`, -/// and steps again. Once the run has decided -/// its outcome ([`Run::decided`]), the effects it still issues are -/// answered [`EffectAnswer::Dropped`] without reaching `perform`, as a -/// caller abandoning a cancelled run would answer them. +/// and steps again. An effect the step also lists as withdrawn is answered +/// [`EffectAnswer::Dropped`] without reaching `perform`, as the run asks +/// of its caller. /// /// `perform` is handed the effect's id beside the effect so a scripted /// performer can correlate answers however it likes; it must return an @@ -116,12 +113,12 @@ pub fn perform_locally( /// # Panics /// /// Only when `perform` panics. An answer of the wrong kind ends the run -/// with an internal failure in its result, and so does a pending step +/// with an internal failure in its outcome, and so does a pending step /// that issued no effect. pub fn drive( mut run: Run, mut perform: impl FnMut(EffectId, &Effect) -> EffectAnswer, -) -> (RunResult, Vec) { +) -> (RunOutcome, Vec) { let mut reported = Vec::new(); let mut answer = |id: EffectId, effect: &Effect| Some(perform(id, effect)); loop { @@ -133,8 +130,8 @@ pub fn drive( &mut Vec::new(), &mut reported, ); - if let Some(result) = stepped { - return (result, reported); + if let Some(outcome) = stepped { + return (outcome, reported); } } } @@ -142,12 +139,13 @@ pub fn drive( /// One step of [`drive`]: steps `run`, appends the step's events to /// `reported`, hands every effect the step issued to `issued` in issue /// order, answers each through `perform`, hands each effect and its answer -/// to `answered`, `Dropped` ones included, and returns the run's result +/// to `answered`, `Dropped` ones included, and returns the run's outcome /// once it has one. /// -/// An effect `perform` answers `None` joins `held`, unanswered. Once the -/// run has decided its outcome, every effect it issues and every held one -/// is answered `Dropped`. +/// An effect `perform` answers `None` joins `held`, unanswered. Every +/// effect the step withdraws, whether it issued the effect or an earlier +/// step did and `held` keeps it, is answered `Dropped`, so the step that +/// decides the run's outcome drops every held effect. pub(crate) fn drive_step( run: &mut Run, issued: &mut impl FnMut(&Effect), @@ -155,14 +153,16 @@ pub(crate) fn drive_step( answered: &mut impl FnMut(&Effect, &EffectAnswer), held: &mut Vec<(EffectId, Effect)>, reported: &mut Vec, -) -> Option { +) -> Option { match run.step() { - Step::Done { result, events } => { + Step::Done { outcome, events } => { reported.extend(events); - Some(result) + Some(outcome) } Step::Pending { - effects, events, .. + effects, + events, + withdrawn, } => { reported.extend(events); if effects.is_empty() && held.is_empty() { @@ -173,12 +173,11 @@ pub(crate) fn drive_step( let error = Error::internal( "the serial driver was handed a pending run with no effect to answer", ); - return Some(RunResult::Failure(RunError::from(error))); + return Some(error.to_outcome()); } - let decided = run.decided(); for IssuedEffect { id, effect, .. } in effects { issued(&effect); - let answer = if decided { + let answer = if withdrawn.contains(&id) { Some(EffectAnswer::Dropped) } else { perform(id, &effect) @@ -191,11 +190,13 @@ pub(crate) fn drive_step( None => held.push((id, effect)), } } - if decided { - for (id, effect) in held.drain(..) { - answered(&effect, &EffectAnswer::Dropped); - run.resume(id, EffectAnswer::Dropped); - } + let (dropped, kept): (Vec<_>, Vec<_>) = std::mem::take(held) + .into_iter() + .partition(|(id, _)| withdrawn.contains(id)); + *held = kept; + for (id, effect) in dropped { + answered(&effect, &EffectAnswer::Dropped); + run.resume(id, EffectAnswer::Dropped); } None } diff --git a/crates/runvil-internal/engine/src/test_support/recording.rs b/crates/runvil-internal/engine/src/test_support/recording.rs index 484584890..60a21284d 100644 --- a/crates/runvil-internal/engine/src/test_support/recording.rs +++ b/crates/runvil-internal/engine/src/test_support/recording.rs @@ -135,11 +135,7 @@ pub(crate) struct ThinkingReport<'a> { #[cfg(test)] #[must_use] pub(crate) fn null_emitter() -> runvil_types::emitter::Emitter { - runvil_types::emitter::Emitter::root( - runvil_types::emitter::EventSink::default(), - "test", - runvil_types::emitter::DebugMode::Off, - ) + runvil_types::emitter::Emitter::root(runvil_types::emitter::EventSink::default(), "test", false) } /// An [`Observer`] that discards every report: what a suite installs when diff --git a/crates/runvil-internal/engine/src/test_support/recording/forward-tests-variants.rs b/crates/runvil-internal/engine/src/test_support/recording/forward-tests-variants.rs index 4513fc291..091f16e05 100644 --- a/crates/runvil-internal/engine/src/test_support/recording/forward-tests-variants.rs +++ b/crates/runvil-internal/engine/src/test_support/recording/forward-tests-variants.rs @@ -73,17 +73,11 @@ fn one_of_every_event_variant() -> Vec<(Event, Seam)> { LuaChunkStarted, LuaChunkSucceeded, LuaChunkFailed, - LuaReplyBindingStarted, - LuaReplyBindingSucceeded, - LuaReplyBindingFailed, LuaTeardownStarted, LuaTeardownSucceeded, ToolScopeValidationStarted, ToolScopeValidationSucceeded, ToolScopeValidationFailed, - ModelCatalogValidationStarted, - ModelCatalogValidationSucceeded, - ModelCatalogValidationFailed, VfsWriteSucceeded, VfsWriteFailed, VfsAppendSucceeded, @@ -200,25 +194,6 @@ fn one_of_every_event_variant() -> Vec<(Event, Seam)> { }, Seam::Observed, ), - ( - Event::TaskResumed { - execution: "run".to_owned(), - section: "A".to_owned(), - provenance: provenance(), - task: task.clone(), - }, - Seam::Observed, - ), - ( - Event::TaskNote { - execution: "run".to_owned(), - section: "A".to_owned(), - provenance: provenance(), - task: task.clone(), - text: "halfway".to_owned(), - }, - Seam::Observed, - ), ( Event::Thinking { execution: "run".to_owned(), diff --git a/crates/runvil-internal/engine/src/test_support/recording/forward.rs b/crates/runvil-internal/engine/src/test_support/recording/forward.rs index 0c55ed945..f2c35123c 100644 --- a/crates/runvil-internal/engine/src/test_support/recording/forward.rs +++ b/crates/runvil-internal/engine/src/test_support/recording/forward.rs @@ -49,17 +49,11 @@ lifecycle_pairs! { LuaChunkStarted, LuaChunkSucceeded, LuaChunkFailed, - LuaReplyBindingStarted, - LuaReplyBindingSucceeded, - LuaReplyBindingFailed, LuaTeardownStarted, LuaTeardownSucceeded, ToolScopeValidationStarted, ToolScopeValidationSucceeded, ToolScopeValidationFailed, - ModelCatalogValidationStarted, - ModelCatalogValidationSucceeded, - ModelCatalogValidationFailed, VfsWriteSucceeded, VfsWriteFailed, VfsAppendSucceeded, @@ -92,8 +86,6 @@ macro_rules! task_lifecycle_variants { | Event::TaskFailed { .. } | Event::TaskCancelled { .. } | Event::TaskAbandoned { .. } - | Event::TaskResumed { .. } - | Event::TaskNote { .. } }; } @@ -269,20 +261,6 @@ fn forward_task(event: Event, observer: &dyn Observer) { §ion, Observation::TaskAbandoned { task, reason }, ), - Event::TaskResumed { - execution, section, .. - } => observer.observe( - &execution, - §ion, - Observation::Other("Task resumed".to_owned()), - ), - Event::TaskNote { - execution, section, .. - } => observer.observe( - &execution, - §ion, - Observation::Other("Task note".to_owned()), - ), // Routed here by `forward_one` for this group alone; `Event` is // `#[non_exhaustive]` in `runvil-types`. _ => {} diff --git a/crates/runvil-internal/engine/src/test_support/recording/observation.rs b/crates/runvil-internal/engine/src/test_support/recording/observation.rs index 0ada73aa8..c01bfdd44 100644 --- a/crates/runvil-internal/engine/src/test_support/recording/observation.rs +++ b/crates/runvil-internal/engine/src/test_support/recording/observation.rs @@ -52,17 +52,11 @@ pub(crate) mod detail { LUA_CHUNK_STARTED => LuaChunkStarted, LUA_CHUNK_SUCCEEDED => LuaChunkSucceeded, LUA_CHUNK_FAILED => LuaChunkFailed, - LUA_REPLY_BINDING_STARTED => LuaReplyBindingStarted, - LUA_REPLY_BINDING_SUCCEEDED => LuaReplyBindingSucceeded, - LUA_REPLY_BINDING_FAILED => LuaReplyBindingFailed, LUA_TEARDOWN_STARTED => LuaTeardownStarted, LUA_TEARDOWN_SUCCEEDED => LuaTeardownSucceeded, TOOL_SCOPE_VALIDATION_STARTED => ToolScopeValidationStarted, TOOL_SCOPE_VALIDATION_SUCCEEDED => ToolScopeValidationSucceeded, TOOL_SCOPE_VALIDATION_FAILED => ToolScopeValidationFailed, - MODEL_CATALOG_VALIDATION_STARTED => ModelCatalogValidationStarted, - MODEL_CATALOG_VALIDATION_SUCCEEDED => ModelCatalogValidationSucceeded, - MODEL_CATALOG_VALIDATION_FAILED => ModelCatalogValidationFailed, VFS_WRITE_SUCCEEDED => VfsWriteSucceeded, VFS_WRITE_FAILED => VfsWriteFailed, VFS_APPEND_SUCCEEDED => VfsAppendSucceeded, @@ -147,12 +141,6 @@ pub(crate) enum Observation { LuaChunkSucceeded, /// A section VM failed to execute a Lua chunk. LuaChunkFailed, - /// A section VM began binding a model reply. - LuaReplyBindingStarted, - /// A section VM bound a model reply successfully. - LuaReplyBindingSucceeded, - /// A section VM failed to bind a model reply. - LuaReplyBindingFailed, /// A section VM began teardown. LuaTeardownStarted, /// A section VM completed teardown. @@ -163,12 +151,6 @@ pub(crate) enum Observation { ToolScopeValidationSucceeded, /// A model-visible tool scope failed semantic validation. ToolScopeValidationFailed, - /// Live-catalog model binding validation began. - ModelCatalogValidationStarted, - /// Live-catalog model binding validation succeeded. - ModelCatalogValidationSucceeded, - /// Live-catalog model binding validation failed. - ModelCatalogValidationFailed, /// An effect-mediated store write succeeded. VfsWriteSucceeded, /// An effect-mediated store write failed. @@ -274,17 +256,11 @@ impl Observation { Observation::LuaChunkStarted => "Lua chunk started", Observation::LuaChunkSucceeded => "Lua chunk succeeded", Observation::LuaChunkFailed => "Lua chunk failed", - Observation::LuaReplyBindingStarted => "Lua reply binding started", - Observation::LuaReplyBindingSucceeded => "Lua reply binding succeeded", - Observation::LuaReplyBindingFailed => "Lua reply binding failed", Observation::LuaTeardownStarted => "Lua teardown started", Observation::LuaTeardownSucceeded => "Lua teardown succeeded", Observation::ToolScopeValidationStarted => "Tool scope validation started", Observation::ToolScopeValidationSucceeded => "Tool scope validation succeeded", Observation::ToolScopeValidationFailed => "Tool scope validation failed", - Observation::ModelCatalogValidationStarted => "Model catalog validation started", - Observation::ModelCatalogValidationSucceeded => "Model catalog validation succeeded", - Observation::ModelCatalogValidationFailed => "Model catalog validation failed", Observation::VfsWriteSucceeded => "Vfs write succeeded", Observation::VfsWriteFailed => "Vfs write failed", Observation::VfsAppendSucceeded => "Vfs append succeeded", diff --git a/crates/runvil-internal/engine/src/test_support/rig.rs b/crates/runvil-internal/engine/src/test_support/rig.rs index 2fcf71545..c3a737547 100644 --- a/crates/runvil-internal/engine/src/test_support/rig.rs +++ b/crates/runvil-internal/engine/src/test_support/rig.rs @@ -30,7 +30,7 @@ struct Tap { impl Tap { fn record(&mut self, effect: &Effect) { - self.issued.push(effect.record()); + self.issued.push(effect.to_record()); if let Effect::Chat { round, .. } = effect { self.rounds.push(*round); } @@ -46,8 +46,8 @@ pub(crate) struct Rig { } impl Rig { - /// Parses the scenario's source, prepares its run, and starts it on - /// the rig. + /// Parses the scenario's source, builds its run, and starts it on the + /// rig. /// /// # Panics /// When the source fails to parse, naming the error. @@ -129,22 +129,12 @@ impl Rig { } /// The run itself, for a test that hands it an answer by hand. - /// - /// # Panics - /// When the prepare refused the run. - #[track_caller] pub(crate) fn run_for_test(&mut self) -> &mut Run { - self.driving - .run_mut() - .expect("the prepare accepted the run") + self.driving.run_mut() } /// The scheduler behind the run, for a test that reaches below the run /// between steps. - /// - /// # Panics - /// When the prepare refused the run, which then has no scheduler. - #[track_caller] pub(crate) fn scheduler_for_test(&mut self) -> &mut Scheduler { self.run_for_test().scheduler_for_test() } @@ -153,7 +143,7 @@ impl Rig { /// /// # Panics /// When the run waits only on held effects, which nothing releases - /// until the run decides. + /// until the run withdraws them. #[track_caller] pub(crate) fn finish(mut self) -> RigTrace { self.settle(); @@ -170,13 +160,13 @@ impl Rig { /// rounds it issued in issue order, and the finished run. pub(crate) struct RigTrace { trace: Trace, - /// The finished run, or `None` when the prepare refused it. - run: Option, + /// The finished run. + run: Run, tap: Tap, } impl RigTrace { - /// The run's result, events, answered effects, and store. + /// The run's outcome, events, answered effects, and store. pub(crate) fn trace(&self) -> &Trace { &self.trace } @@ -194,15 +184,7 @@ impl RigTrace { /// The state of `task`'s entry at the run's end, or `None` when no task /// with that id was started. - /// - /// # Panics - /// When the prepare refused the run, which then has no scheduler. - #[track_caller] pub(crate) fn task_state_for_test(&mut self, task: &TaskId) -> Option { - self.run - .as_mut() - .expect("the prepare accepted the run") - .scheduler_for_test() - .task_state_for_test(task) + self.run.scheduler_for_test().task_state_for_test(task) } } diff --git a/crates/runvil-internal/engine/src/test_support/scenario-driving.rs b/crates/runvil-internal/engine/src/test_support/scenario-driving.rs index 7913876ee..ad0e8dbc2 100644 --- a/crates/runvil-internal/engine/src/test_support/scenario-driving.rs +++ b/crates/runvil-internal/engine/src/test_support/scenario-driving.rs @@ -9,6 +9,7 @@ use std::sync::Mutex; use std::sync::mpsc::Sender; use std::thread::JoinHandle; +use runvil_types::cancel::CancelHandle; use runvil_types::event::Event; use runvil_types::tools::{ToolError, ToolId}; use runvil_vfs::{Origin, VfsError, VfsRef}; @@ -19,9 +20,8 @@ use super::super::recording::{ }; use super::super::{drive_step, forward}; use super::{Chat, Matcher}; -use crate::execute::{ - AnswerRecord, Effect, EffectAnswer, EffectId, EffectRecord, Run, RunError, RunResult, -}; +use crate::Error; +use crate::execute::{AnswerRecord, Effect, EffectAnswer, EffectId, EffectRecord, Run, RunOutcome}; use crate::lua::run_store_op; use crate::model::{CompletionError, CompletionErrorKind}; @@ -93,19 +93,22 @@ impl Answers { /// A started scenario, stepped by hand or to its end. pub(crate) struct Driving { - /// The run, kept once it is done; `None` when the prepare refused it. - pub(super) run: Option, - /// The run's result, once it is done or when the prepare refused it. - pub(super) result: Option, + /// The run, kept once it is done. + pub(super) run: Run, + /// The run's outcome, once it is done. + pub(super) outcome: Option, pub(super) answers: Answers, /// The parse events, then every event the run reported, in order. pub(super) events: Vec, /// How many of the events the parse reported. pub(super) parsed: usize, pub(super) effects: Vec<(EffectRecord, AnswerRecord)>, - /// The effects the scenario holds, unanswered until the run decides. + /// The effects the scenario holds, unanswered until a test releases + /// them or the run withdraws them. pub(super) held: Vec<(EffectId, Effect)>, pub(super) store: TestStore, + /// The cancel flag the scenario handed the run's config. + pub(super) cancel: CancelHandle, /// The thread that cancels the run on the scenario's signal. pub(super) canceller: Option>, } @@ -132,19 +135,18 @@ impl Driving { mut issued: impl FnMut(&Effect), order: impl FnOnce(&mut [(EffectId, Effect)]), ) -> bool { - if self.result.is_some() { + if self.outcome.is_some() { return false; } - let Some(run) = self.run.as_mut() else { - return false; - }; + let run = &mut self.run; let effects = &mut self.effects; let mut moved = false; let mut answered = |effect: &Effect, answer: &EffectAnswer| { moved = true; - effects.push((effect.record(), answer.record())); + effects.push((effect.to_record(), answer.to_record())); }; let holding = self.held.len(); + let kept: Vec = self.held.iter().map(|(id, _)| *id).collect(); let stepped = drive_step( run, &mut issued, @@ -153,19 +155,22 @@ impl Driving { &mut self.held, &mut self.events, ); - if let Some(result) = stepped { - self.result = Some(result); + if let Some(outcome) = stepped { + self.outcome = Some(outcome); return false; } - // A deciding step answered every effect, held ones included, so - // only an undecided step leaves a wave to answer. - let mut wave = self.held.split_off(holding.min(self.held.len())); + // The step dropped what it withdrew, so what it left held beyond + // the earlier holds is the wave this step issued. + let (held, mut wave): (Vec<_>, Vec<_>) = std::mem::take(&mut self.held) + .into_iter() + .partition(|(id, _)| kept.contains(id)); + self.held = held; order(&mut wave); for (id, effect) in wave { match self.answers.answer(&effect) { Some(answer) => { moved = true; - self.effects.push((effect.record(), answer.record())); + self.effects.push((effect.to_record(), answer.to_record())); run.resume(id, answer); } None => self.held.push((id, effect)), @@ -186,11 +191,8 @@ impl Driving { }; let (id, effect) = self.held.remove(at); let answer = self.answers.perform(&effect); - self.effects.push((effect.record(), answer.record())); - self.run - .as_mut() - .expect("a run holding an effect has started") - .resume(id, answer); + self.effects.push((effect.to_record(), answer.to_record())); + self.run.resume(id, answer); } /// The parse events, then every event the run reported so far, in @@ -199,9 +201,9 @@ impl Driving { &self.events } - /// The run, done or not, or `None` when the prepare refused it. - pub(crate) fn run_mut(&mut self) -> Option<&mut Run> { - self.run.as_mut() + /// The run, done or not. + pub(crate) fn run_mut(&mut self) -> &mut Run { + &mut self.run } /// Joins the canceller once it is woken, so the next step sees its @@ -223,10 +225,8 @@ impl Driving { /// Sets the run's cancel flag between steps, as a caller cancelling a /// run it steps would. A run that is done ignores it. pub(crate) fn cancel(&mut self) { - if self.result.is_none() - && let Some(run) = self.run.as_mut() - { - run.cancel(); + if self.outcome.is_none() { + self.cancel.cancel(); } } @@ -234,7 +234,7 @@ impl Driving { /// /// # Panics /// When the run waits only on held effects, which nothing releases - /// until the run decides. + /// until the run withdraws them. #[track_caller] pub(crate) fn finish(mut self) -> Trace { while self.step() {} @@ -242,17 +242,16 @@ impl Driving { } /// Ends a run already stepped as far as it goes and returns what it - /// produced, beside the run itself, or `None` when the prepare refused - /// it. + /// produced, beside the run itself. /// /// # Panics /// When the run is not done: it waits only on held effects, which - /// nothing releases until the run decides. + /// nothing releases until the run withdraws them. #[track_caller] - pub(crate) fn end(self) -> (Trace, Option) { - let Some(result) = self.result else { + pub(crate) fn end(mut self) -> (Trace, Run) { + let Some(outcome) = self.outcome else { panic!( - "the run waits only on {} held effects, and no cancel has decided it", + "the run waits only on {} held effects, and no cancel has withdrawn them", self.held.len() ); }; @@ -265,21 +264,24 @@ impl Driving { .expect("the canceller thread does not panic"); } let trace = Trace { - result, + outcome, + failure: self.run.take_failure_for_test(), events: self.events, parsed: self.parsed, effects: self.effects, store: self.store, - turns: self.run.as_ref().map_or(0, Run::turns_for_test), + turns: self.run.turns_for_test(), }; (trace, self.run) } } -/// What a scenario's run produced: its result, its events, every effect -/// it issued paired with its answer, its store, and its turn count. +/// What a scenario's run produced: its outcome, the Engine error behind a +/// failed one, its events, every effect it issued paired with its answer, +/// its store, and its turn count. pub(crate) struct Trace { - result: RunResult, + outcome: RunOutcome, + failure: Option, /// The parse events, then every event the run reported, in order. events: Vec, /// How many of the events the parse reported. @@ -290,32 +292,32 @@ pub(crate) struct Trace { } impl Trace { - /// The run's result. - pub(crate) fn result(&self) -> &RunResult { - &self.result + /// The run's outcome. + pub(crate) fn outcome(&self) -> &RunOutcome { + &self.outcome } /// The text of a run that succeeded. /// /// # Panics - /// When the run did not succeed, naming its result. + /// When the run did not succeed, naming its outcome. #[track_caller] pub(crate) fn text(&self) -> &str { - match &self.result { - RunResult::Completed(text) => text, + match &self.outcome { + RunOutcome::Completed { final_text } => final_text, other => panic!("expected the run to succeed, got {other:?}"), } } - /// The error of a run that failed. + /// The Engine error behind the outcome of a run that failed. /// /// # Panics - /// When the run did not fail, naming its result. + /// When the run did not fail, naming its outcome. #[track_caller] - pub(crate) fn error(&self) -> &RunError { - match &self.result { - RunResult::Failure(error) => error, - other => panic!("expected the run to fail, got {other:?}"), + pub(crate) fn error(&self) -> &Error { + match (&self.outcome, &self.failure) { + (RunOutcome::Failed { .. }, Some(error)) => error, + (other, _) => panic!("expected the run to fail, got {other:?}"), } } @@ -332,8 +334,8 @@ impl Trace { /// Every effect the run issued, beside its answer, in the order the /// answers went, which is issue order for every effect the scenario - /// does not hold. An effect issued after the run decided its outcome, - /// and a held one, is paired with [`AnswerRecord::Dropped`]. + /// does not hold. An effect the run withdrew before the scenario + /// answered it is paired with [`AnswerRecord::Dropped`]. pub(crate) fn effects(&self) -> &[(EffectRecord, AnswerRecord)] { &self.effects } @@ -355,7 +357,7 @@ impl Trace { } /// The debug pair of every model turn the run reported, request before - /// response, in order. The run reports the pair only when its context + /// response, in order. The run reports the pair only when its config /// turns debug reporting on. pub(crate) fn captures(&self) -> Vec { let captures = Captures::default(); @@ -400,7 +402,7 @@ impl TestStore { TestStore(Mutex::new(vfs)) } - /// The handle the run and the context builders take: the run's whole + /// The handle the run and the config builders take: the run's whole /// filesystem, store included. pub(crate) fn vfs(&self) -> VfsRef { self.0 diff --git a/crates/runvil-internal/engine/src/test_support/scenario-serial.rs b/crates/runvil-internal/engine/src/test_support/scenario-serial.rs index 921d4c0d5..8f51640fa 100644 --- a/crates/runvil-internal/engine/src/test_support/scenario-serial.rs +++ b/crates/runvil-internal/engine/src/test_support/scenario-serial.rs @@ -10,7 +10,7 @@ use runvil_types::ids::{Provenance, TaskId}; use super::super::perform_locally; use super::super::scripted_chat::{ScriptedReply, text}; use crate::execute::{ - Effect, EffectAnswer, EffectId, EffectRecord, IssuedEffect, Run, RunResult, Step, + Effect, EffectAnswer, EffectId, EffectRecord, IssuedEffect, Run, RunOutcome, Step, }; /// The answer a `Chat` effect gets from the scripted model's `reply`, @@ -64,7 +64,7 @@ pub(crate) fn round_trip(record: &EffectRecord) -> EffectRecord { /// prompt a `models.infer` round sends. #[track_caller] pub(crate) fn infer_prompt(effect: &Effect) -> String { - let EffectRecord::Chat { messages, .. } = effect.record() else { + let EffectRecord::Chat { messages, .. } = effect.to_record() else { panic!("a chat effect records its messages: {effect:?}"); }; messages[0]["content"] @@ -75,9 +75,9 @@ pub(crate) fn infer_prompt(effect: &Effect) -> String { /// The text of a run that succeeded; panics on any other outcome. #[track_caller] -pub(crate) fn text_of(result: RunResult) -> String { - match result { - RunResult::Completed(text) => text, +pub(crate) fn text_of(outcome: RunOutcome) -> String { + match outcome { + RunOutcome::Completed { final_text } => final_text, other => panic!("the run succeeds: {other:?}"), } } @@ -101,7 +101,7 @@ pub(crate) enum Batching { /// Everything one driven run produced, in the forms the properties /// compare. pub(crate) struct Outcome { - pub(crate) result: RunResult, + pub(crate) result: RunOutcome, pub(crate) events: Vec, pub(crate) effects: Vec<(Provenance, EffectRecord)>, } @@ -130,7 +130,7 @@ impl Outcome { #[track_caller] pub(crate) fn text(&self) -> &str { match &self.result { - RunResult::Completed(text) => text, + RunOutcome::Completed { final_text } => final_text, other => panic!("the run succeeds: {other:?}"), } } @@ -145,12 +145,12 @@ pub(crate) fn drive_batched(mut run: Run, batching: Batching) -> Outcome { loop { match run.step() { Step::Done { - result, + outcome, events: more, } => { events.extend(more); return Outcome { - result, + result: outcome, events, effects, }; @@ -167,7 +167,7 @@ pub(crate) fn drive_batched(mut run: Run, batching: Batching) -> Outcome { effect, } in issued { - effects.push((provenance, effect.record())); + effects.push((provenance, effect.to_record())); outstanding.push((id, effect)); } assert!( diff --git a/crates/runvil-internal/engine/src/test_support/scenario.rs b/crates/runvil-internal/engine/src/test_support/scenario.rs index 02b918d83..ba796e321 100644 --- a/crates/runvil-internal/engine/src/test_support/scenario.rs +++ b/crates/runvil-internal/engine/src/test_support/scenario.rs @@ -1,7 +1,7 @@ //! The run-level test driver. A [`Scenario`] holds a run's setup as data: -//! its source, args, run name, seed, context edits, store, tools, model, +//! its source, args, run name, seed, config edits, store, tools, model, //! the sets it fills in before the run starts, the effects it holds, and -//! the effect that cancels it. It parses the source, prepares the run, and +//! the effect that cancels it. It parses the source, builds the run, and //! steps it on the calling thread, answering every effect it does not hold //! the moment it is issued, and returns a [`Trace`] of what the run //! produced. Nothing awaits or sleeps: a timer fires at once, a tool is a @@ -13,6 +13,7 @@ //! effects, its store, and its trace in `driving`. use std::cell::Cell; +use std::collections::BTreeMap; use std::num::NonZeroU32; use std::sync::Arc; use std::sync::mpsc; @@ -20,6 +21,7 @@ use std::thread; use runvil_lua::ToolOutputKind; use runvil_model_client::model::ModelInvocation; +use runvil_types::cancel::CancelHandle; use runvil_types::event::Event; use runvil_types::seed::RunSeed; use runvil_types::timestamp::Timestamp; @@ -29,9 +31,11 @@ use runvil_vfs::VfsRef; use super::FnTool; use super::scripted_chat::ScriptedChat; use crate::Error; -use crate::execute::{Effect, EffectAnswer, Run, RunContext, RunError, RunLimits, RunResult}; +use crate::execute::{Effect, EffectAnswer, Run, RunConfig, RunLimits}; use crate::lua::ToolBinding; -use crate::model::{CompletionErrorKind, ModelBinding, ModelId, ModelInfo, ModelSet, ThinkingMode}; +use crate::model::{ + CompletionErrorKind, ModelBinding, ModelId, ModelInfo, ModelRole, ModelSet, ThinkingMode, +}; use crate::parser::Prompt; #[path = "scenario-driving.rs"] @@ -51,11 +55,11 @@ pub(crate) const EXECUTION: &str = "execute-test"; /// The fixed caller inputs every test run shares: a seed and a start instant /// a test that does not care about them never has to choose. The tests of -/// the inputs themselves (`run_inputs`) build their contexts directly. +/// the inputs themselves (`run_inputs`) build their configs directly. pub(crate) const TEST_SEED: RunSeed = RunSeed::from_bytes([1; 32]); pub(crate) const TEST_STARTED_AT: Timestamp = Timestamp::from_unix_millis(1_700_000_000_000); -/// The gateway model a bound scenario's roles resolve to. +/// The gateway model a bound scenario's `default` role resolves to. pub(crate) const TEST_MODEL: &str = "claude-sonnet-4-6"; /// The gateway model `id` as the tests bind it: a 131072-token window and @@ -93,8 +97,8 @@ pub(crate) fn test_binding(alias: &str, description: &str, model: &str) -> Model /// Asserts `error` is a model failure of `kind` whose message is `message`. #[track_caller] -pub(crate) fn assert_model_failure(error: &RunError, kind: CompletionErrorKind, message: &str) { - let Error::Completion(failure) = error.inner() else { +pub(crate) fn assert_model_failure(error: &Error, kind: CompletionErrorKind, message: &str) { + let Error::Completion(failure) = error else { panic!("expected a model failure, got {error:?}"); }; assert_eq!( @@ -103,8 +107,9 @@ pub(crate) fn assert_model_failure(error: &RunError, kind: CompletionErrorKind, ); } -/// The edit a scenario applies to its run's context before the prepare. -type ContextEdit = Box RunContext>; +/// The edit a scenario applies to its run's config before the run is +/// built. +type ConfigEdit = Box RunConfig>; /// A test over an effect: which effects a scenario holds, or which answer /// signals its canceller. @@ -126,7 +131,7 @@ pub(crate) struct Scenario { args: String, name: String, seed: RunSeed, - context: ContextEdit, + config: ConfigEdit, store: TestStore, /// The catalog's tools. tools: Vec, @@ -136,24 +141,23 @@ pub(crate) struct Scenario { chat: Option, hold: Option, signal: Option, - prepare: bool, - /// The current model the prepare binds every declared role to. - model: Option, - bind_roles: bool, + /// The role map the run fills its model set from. + roles: BTreeMap, + /// The cancel flag the run's config takes, which the scenario keeps. + cancel: CancelHandle, } impl Scenario { /// A run of `source` named [`EXECUTION`], under [`TEST_SEED`] and /// [`TEST_STARTED_AT`], with empty args, a fresh memory store, no tools, - /// no model, nothing filled in, and no held effect, prepared so that - /// an unsatisfiable prompt is refused before it starts. + /// no model, nothing filled in, and no held effect. pub(crate) fn new(source: impl Into) -> Scenario { Scenario { source: source.into(), args: String::new(), name: EXECUTION.to_owned(), seed: TEST_SEED, - context: Box::new(|ctx| ctx), + config: Box::new(|ctx| ctx), store: TestStore::new(), tools: Vec::new(), listed: Vec::new(), @@ -161,9 +165,8 @@ impl Scenario { chat: None, hold: None, signal: None, - prepare: true, - model: None, - bind_roles: false, + roles: BTreeMap::new(), + cancel: CancelHandle::new(), } } @@ -185,41 +188,42 @@ impl Scenario { self } - /// Makes the [`TEST_MODEL`] the run's current model, so the prepare's - /// trivial fill binds every role the prompt declares to it. + /// Makes the [`TEST_MODEL`] the run's `default` model, so the run + /// fills all five roles with it. pub(crate) fn bound(self) -> Scenario { self.current_model(test_model(TEST_MODEL)) } - /// Makes `model` the run's current model, which the prepare binds - /// every role the prompt declares to. - pub(crate) fn current_model(mut self, model: ModelInfo) -> Scenario { - self.model = Some(model); - self + /// Makes `model` the run's `default` model, which fills every role the + /// map gives no model of its own. + pub(crate) fn current_model(self, model: ModelInfo) -> Scenario { + self.role_model(ModelRole::Default, model) } - /// Binds every role the prompt declares to the [`TEST_MODEL`] in the - /// context's model bindings, as the prepare's fill would, without making - /// it the current model, so the run's model set starts empty for the - /// H1 pass to fill. - pub(crate) fn bind_roles(mut self) -> Scenario { - self.bind_roles = true; + /// Gives `role` its own model in the role map the run fills its model + /// set from. + pub(crate) fn role_model(mut self, role: ModelRole, model: ModelInfo) -> Scenario { + self.roles.insert(role, model); self } /// Runs under `limits` in place of the default limits. pub(crate) fn limits(self, limits: RunLimits) -> Scenario { - self.context(move |ctx| ctx.with_limits(limits)) + self.config(move |ctx| ctx.with_limits(limits)) } - /// Adds `edit` to the edits applied to the run's context, after the - /// scenario's own settings and before the prepare. - pub(crate) fn context( + /// Adds `edit` to the edits applied to the run's config, after the + /// scenario's own settings, its role map and cancel flag included. An + /// edit that replaces the cancel flag leaves [`cancel_when`] and + /// [`Driving::cancel`] setting a flag the run never reads. + /// + /// [`cancel_when`]: Scenario::cancel_when + pub(crate) fn config( mut self, - edit: impl FnOnce(RunContext) -> RunContext + 'static, + edit: impl FnOnce(RunConfig) -> RunConfig + 'static, ) -> Scenario { - let earlier = self.context; - self.context = Box::new(move |ctx| edit(earlier(ctx))); + let earlier = self.config; + self.config = Box::new(move |ctx| edit(earlier(ctx))); self } @@ -229,13 +233,6 @@ impl Scenario { self } - /// Starts the run over the context as given, with no prepare: no model - /// role is bound and no requirement is checked. - pub(crate) fn unprepared(mut self) -> Scenario { - self.prepare = false; - self - } - /// Adds `tool` to the run's catalog and answers its calls with it. pub(crate) fn tool(mut self, tool: FnTool) -> Scenario { self.tools.push(tool); @@ -243,7 +240,7 @@ impl Scenario { } /// Fills the run's model set with `set` before the run starts, as the - /// H1 pass would leave it, in place of the set the prepare binds. + /// H1 pass would leave it, in place of the set the role map fills. pub(crate) fn models(mut self, set: ModelSet) -> Scenario { self.prefill.models = Some(set); self @@ -298,8 +295,9 @@ impl Scenario { } /// Leaves every effect `held` matches unanswered while the run keeps - /// stepping, so a cancel can land between steps. Once the run decides - /// its outcome, each held effect is answered `Dropped`. + /// stepping, so a cancel can land between steps. Each held effect the + /// run withdraws is answered `Dropped`, so the step that decides the + /// run's outcome drops them all. pub(crate) fn hold(mut self, held: impl Fn(&Effect) -> bool + 'static) -> Scenario { self.hold = Some(Box::new(held)); self @@ -319,13 +317,13 @@ impl Scenario { self } - /// Parses the source, prepares the run, and starts it. + /// Parses the source, builds the run, and starts it. /// /// # Panics /// When the source fails to parse, naming the error. #[track_caller] pub(crate) fn start(mut self) -> Driving { - let (prompt, events, ctx, refused) = self.prepared(); + let (prompt, events, ctx) = self.config_of(); let (signal, signalled) = match self.signal { Some(signal) => { let (sender, receiver) = mpsc::channel(); @@ -344,11 +342,9 @@ impl Scenario { signal, signalled: Cell::new(false), }; - let run = refused - .is_none() - .then(|| self.prefill.start(Arc::new(prompt), &self.args, ctx)); - let canceller = run.as_ref().zip(signalled).map(|(run, signalled)| { - let cancel = run.cancel_handle(); + let run = self.prefill.start(Arc::new(prompt), &self.args, ctx); + let canceller = signalled.map(|signalled| { + let cancel = self.cancel.clone(); thread::spawn(move || { if signalled.recv().is_ok() { cancel.cancel(); @@ -357,13 +353,14 @@ impl Scenario { }); Driving { run, - result: refused, + outcome: None, answers, parsed: events.len(), events, effects: Vec::new(), held: Vec::new(), store: self.store, + cancel: self.cancel, canceller, } } @@ -377,60 +374,45 @@ impl Scenario { self.start().finish() } - /// Parses the source, prepares the run, and hands it back unstarted, - /// for a test that steps it and answers its effects by hand. The + /// Parses the source, builds the run, and hands it back unstarted, for + /// a test that steps it and answers its effects by hand. The /// scenario's tools still shape the catalog, but its answers, holds, /// and canceller play no part. /// /// # Panics - /// When the source fails to parse or the prepare refuses the run, - /// naming the error. + /// When the source fails to parse, naming the error. #[track_caller] pub(crate) fn into_run(mut self) -> Run { - let (prompt, _, ctx, refused) = self.prepared(); - if let Some(refused) = refused { - panic!("expected the prepare to accept the run, got {refused:?}"); - } + let (prompt, _, ctx) = self.config_of(); self.prefill.start(Arc::new(prompt), &self.args, ctx) } - /// Parses the source and builds the run's context, prepared unless the - /// scenario skips the prepare, beside the parse events and the - /// prepare's refusal. + /// Parses the source and builds the run's config, beside the parse + /// events. #[track_caller] - fn prepared(&mut self) -> (Prompt, Vec, RunContext, Option) { + fn config_of(&mut self) -> (Prompt, Vec, RunConfig) { let (parsed, events) = Prompt::parse(&self.source, &self.name); let prompt = match parsed { Ok(prompt) => prompt, Err(error) => panic!("expected the scenario source to parse, got {error}"), }; - let mut ctx = RunContext::new(self.name.clone(), self.seed, TEST_STARTED_AT) - .with_vfs(self.store.vfs()); + let mut ctx = RunConfig::new(self.name.clone(), self.seed, TEST_STARTED_AT) + .with_vfs(self.store.vfs()) + .with_models(std::mem::take(&mut self.roles)) + .with_cancel(self.cancel.clone()); if !self.tools.is_empty() || !self.listed.is_empty() { let infos = self.tools.iter().map(FnTool::info); let catalog = ToolCatalog::new(infos.chain(self.listed.iter().cloned())) .expect("the scenario's tool ids differ"); ctx = ctx.with_tools(catalog); } - let edit = std::mem::replace(&mut self.context, Box::new(|ctx| ctx)); - let mut ctx = edit(ctx); - if self.bind_roles { - for (label, _) in prompt.frontmatter().models() { - ctx.bind_role_for_test(label, test_model(TEST_MODEL)); - } - } - let mut refused = None; - if self.prepare { - let requirements; - (ctx, requirements) = ctx.prepare(&prompt, self.model.take()); - refused = requirements.refusal().map(RunResult::Failure); - } - (prompt, events, ctx, refused) + let edit = std::mem::replace(&mut self.config, Box::new(|ctx| ctx)); + (prompt, events, edit(ctx)) } } /// What a scenario fills into its run's state before the run starts, in -/// place of what the prepare and the H1 pass would leave there. +/// place of what the role map and the H1 pass would leave there. #[derive(Default)] struct Prefill { models: Option, @@ -440,7 +422,7 @@ struct Prefill { impl Prefill { /// Builds the run of `prompt` under `config` with the state filled in, /// or as [`Run::new`] builds it when there is nothing to fill. - fn start(self, prompt: Arc, args: &str, config: RunContext) -> Run { + fn start(self, prompt: Arc, args: &str, config: RunConfig) -> Run { if self.models.is_none() && !self.raw_shims { return Run::new(prompt, args, config); } diff --git a/crates/runvil-internal/engine/src/test_support/scripted_chat.rs b/crates/runvil-internal/engine/src/test_support/scripted_chat.rs index b8f5a3222..25bdcb4c5 100644 --- a/crates/runvil-internal/engine/src/test_support/scripted_chat.rs +++ b/crates/runvil-internal/engine/src/test_support/scripted_chat.rs @@ -252,6 +252,15 @@ impl ScriptedChat { .clone() } + /// The model each recorded round named, in the order the rounds arrived. + #[must_use] + pub(crate) fn requested_models(&self) -> Vec { + self.requests() + .iter() + .map(|body| body.options.model().to_owned()) + .collect() + } + /// The most recently recorded round, if any. #[must_use] pub(crate) fn last_request(&self) -> Option { diff --git a/crates/runvil-internal/engine/src/test_support/tools.rs b/crates/runvil-internal/engine/src/test_support/tools.rs index 15d9cea61..c06ee7e44 100644 --- a/crates/runvil-internal/engine/src/test_support/tools.rs +++ b/crates/runvil-internal/engine/src/test_support/tools.rs @@ -26,6 +26,7 @@ pub(crate) struct FnTool { schema: serde_json::Value, structured: bool, describes_tools: bool, + model_only_call: bool, handler: Arc, } @@ -46,6 +47,7 @@ impl FnTool { schema, structured: false, describes_tools: false, + model_only_call: false, handler: Arc::new(handler), } } @@ -77,6 +79,13 @@ impl FnTool { self } + /// Marks the tool as one only a model may call, so a script's + /// `tools.call` refuses it. + pub(crate) fn model_only_call(mut self) -> FnTool { + self.model_only_call = true; + self + } + /// The tool's stable identity: the catalog key and what a `ToolCall` /// effect names. pub(crate) fn id(&self) -> ToolId { @@ -88,7 +97,8 @@ impl FnTool { pub(crate) fn info(&self) -> ToolInfo { let info = ToolInfo::new(self.id(), &self.description, self.schema.clone()) .structured(self.structured) - .describes_tools(self.describes_tools); + .describes_tools(self.describes_tools) + .model_only_call(self.model_only_call); match &self.wire_name { Some(wire_name) => info.wire_name(wire_name), None => info, diff --git a/crates/runvil-internal/lua/benches/surface.rs b/crates/runvil-internal/lua/benches/surface.rs index 2db89e4de..a591a1ae9 100644 --- a/crates/runvil-internal/lua/benches/surface.rs +++ b/crates/runvil-internal/lua/benches/surface.rs @@ -15,7 +15,7 @@ )] use std::num::NonZeroU32; -use std::sync::{Arc, Mutex}; +use std::sync::Arc; use criterion::{BatchSize, Criterion, criterion_group, criterion_main}; use runvil_lua::{ @@ -23,7 +23,7 @@ use runvil_lua::{ project_messages, }; use runvil_model_client::model::ModelSet; -use runvil_types::emitter::{DebugMode, Emitter, EventSink}; +use runvil_types::emitter::{Emitter, EventSink}; use runvil_types::seed::RunSeed; use runvil_types::untrusted::GuardNonce; use serde_json::json; @@ -33,7 +33,7 @@ const SECTION: &str = "Bench"; /// An emitter over a sink nobody drains: the bench measures the VM, not /// the reports. fn emitter() -> Emitter { - Emitter::root(EventSink::default(), "bench", DebugMode::Off) + Emitter::root(EventSink::default(), "bench", false) } /// A section VM with Engine values injected, so the section's `messages` @@ -42,7 +42,7 @@ fn builder_vm() -> SectionVm { SectionVm::new_for_section( &GuardNonce::from_seed(&RunSeed::from_bytes([1; 32])), &Arc::new(ToolSet::default()), - &Arc::new(Mutex::new(ModelSet::default())), + &Arc::new(ModelSet::default()), &emitter(), SECTION, ) diff --git a/crates/runvil-internal/lua/src/__impl_coro.lua b/crates/runvil-internal/lua/src/__impl_coro.lua index b30914e75..d7dffe23c 100644 --- a/crates/runvil-internal/lua/src/__impl_coro.lua +++ b/crates/runvil-internal/lua/src/__impl_coro.lua @@ -18,9 +18,10 @@ -- run's cancel flag is set: a failure caught under cancellation is the -- instruction hook's abort, which must unwind to the block guard, so every -- protected call below raises it again instead of returning it, --- `is_model_handle(value)` reports whether a value is a model handle, --- `loop_begin(entry, ...)` starts one `models.loop` call in Rust over the --- run's round cap, the section's `compactors` table, and the section's +-- `is_model_handle(value)` reports whether a value is a model handle, so +-- the `infer` and `loop` entries can read a leading model handle before +-- any yield, `loop_begin(...)` starts one `models.loop` call in Rust over +-- the run's round cap, the section's `compactors` table, and the section's -- `messages` list, returning the -- call's step function and its first action, and `compact_begin(...)` -- starts one `models.compact` call over the same list the same way. The @@ -118,34 +119,24 @@ local function run_infer(handle, prompt) return result end --- models.infer(prompt) and a model handle's `infer`, as `h:infer(prompt)`. --- Handles carry `infer` and `loop` as fields that read these entries, Lua --- functions because each may suspend and a Rust method cannot. Each entry --- checks its receiver and then its arity, counting trailing nils, before --- any yield. +-- models.infer(prompt) runs on the section's current model, and +-- models.infer(model, prompt) runs on the model handle's frozen binding. +-- The leading handle is checked first, then the arity for that shape is +-- counted with trailing nils, before any yield. local function infer(...) if is_model_handle((...)) then - raise("lua", { - message = "models.infer takes (prompt); call handle:infer(prompt) to run on a model handle", - }) + if select('#', ...) > 2 then + raise("lua", { message = "models.infer takes (prompt) or (model, prompt)" }) + end + local handle, prompt = ... + return run_infer(handle, prompt) end if select('#', ...) > 1 then - raise("lua", { message = "models.infer takes (prompt)" }) + raise("lua", { message = "models.infer takes (prompt) or (model, prompt)" }) end return run_infer(nil, (...)) end -local function handle_infer(...) - if not is_model_handle((...)) then - raise("lua", { message = "call infer on a model handle with a colon: handle:infer(prompt)" }) - end - if select('#', ...) > 2 then - raise("lua", { message = "handle:infer takes (prompt)" }) - end - local handle, prompt = ... - return run_infer(handle, prompt) -end - local function call_section(target, input) local ok, result = yield({ op = "call", @@ -250,17 +241,11 @@ local function drive(step, action, a, b) end -- models.loop(compactor?) runs the section's `messages` list on the --- section's current model, and a model handle's `loop`, as --- `h:loop(compactor?)`, runs every round on the handle's frozen binding. --- The Engine installs the --- first as models.loop. Each entry names itself to `loop_begin`, so the --- argument checks know the form without inspecting the arguments. +-- section's current model, and models.loop(model, compactor?) runs every +-- round on the model handle's frozen binding. The compactor is optional in +-- both shapes. The Engine installs this as models.loop. local function models_loop(...) - return drive(loop_begin(false, ...)) -end - -local function handle_loop(...) - return drive(loop_begin(true, ...)) + return drive(loop_begin(...)) end -- models.compact(compactor, reason) runs `compactor(reason)` once over the @@ -356,8 +341,6 @@ return { helpers = { raise = raise, fail = fail, engine_type = engine_type }, loop = models_loop, compact = models_compact, - -- The fields a model handle reads, through the registry stash. - handle_methods = { infer = handle_infer, loop = handle_loop }, model_tool_call = tools_call_as_model, guard = guard, pcall = protected_call, diff --git a/crates/runvil-internal/lua/src/__impl_fanout.lua b/crates/runvil-internal/lua/src/__impl_fanout.lua index 832cf9914..c73fd4188 100644 --- a/crates/runvil-internal/lua/src/__impl_fanout.lua +++ b/crates/runvil-internal/lua/src/__impl_fanout.lua @@ -13,15 +13,20 @@ -- `collection_members(collection)` enumerates a collection as `(members)` -- or `(nil, message)` - the array part in order, then the hash part as -- `{ key, value }` pairs sorted by key - and `render_item(item)` renders a --- member as `{{ item }}` would. +-- member as `{{ item }}` would. The last three captures reach the chain's +-- current model: `get_selection()` returns its handle, or nil when the +-- caller supplied no model, `set_selection(handle_or_nil)` sets it (nil +-- clears it) and refuses any other value, and `is_model_handle(value)` +-- reports whether a value is a model handle. local yield, var_snapshot, helpers, - collection_members, render_item = ... + collection_members, render_item, + get_selection, set_selection, is_model_handle = ... local raise, fail = helpers.raise, helpers.fail --- setmetatable, captured at install so the shim's behavior does not move --- when author code rebinds the base globals. -local setmetatable = setmetatable +-- The base functions the shim uses, captured at install so its behavior does +-- not move when author code rebinds the base globals. +local setmetatable, type, select = setmetatable, type, select -- One fanout arm's result: a frozen, methodless object whose fields -- are `text`, `ok`, `item`, and `exhausted`, with `tostring` giving the @@ -51,8 +56,16 @@ local function exhausted_stub(item) return "## " .. render_item(item) .. "\n\nUNKNOWN\n\n(section incomplete: tool loop exhausted)" end --- tasks.fanout(worker, collection): run `worker` once per collection --- member as a task chain and return the results in collection order. The +-- tasks.fanout(worker, collection) and tasks.fanout(model, worker, +-- collection): run `worker` once per collection member as a task chain and +-- return the results in collection order. The model-first form takes a +-- model handle first and runs every arm on it: it sets the chain's +-- current model to the handle before the spawns, so each arm copies it when +-- it is dispatched, and puts the saved model back before the first join and +-- before a spawn failure is re-raised. No author code runs in the parent +-- between the switch and the restore, and a model switch is not an event, +-- so the switch cannot be observed. The first argument is checked before +-- anything is switched. The -- members enumerate (array part in order, then the hash part as -- `{ key, value }` pairs sorted by key), an empty collection raises before -- any spawn, and every arm is spawned up front - before any join - so no @@ -68,7 +81,23 @@ end -- failure cancels the live arms and re-raises. No arm outlives the call: -- every arm is delivered or cancelled before the function returns or -- raises. -local function fanout(worker, collection) +local function fanout(...) + local argc = select("#", ...) + local model, worker, collection + if argc == 2 then + worker, collection = ... + elseif argc == 3 then + model, worker, collection = ... + if not is_model_handle(model) then + raise("lua", { + message = "tasks.fanout: the first argument must be a model handle, got " .. type(model), + }) + end + else + raise("lua", { + message = "tasks.fanout takes (worker, collection) or (model, worker, collection)", + }) + end local members, message = collection_members(collection) if not members then raise("lua", { message = message }) @@ -95,6 +124,21 @@ local function fanout(worker, collection) live, slot_of = {}, {} end + -- The model-first form switches the chain's current model for the spawns + -- and puts it back once, whichever path gets there first. + local switched, saved = false, nil + if model ~= nil then + saved = get_selection() + set_selection(model) + switched = true + end + local function restore() + if switched then + switched = false + set_selection(saved) + end + end + -- Every arm is spawned up front, before the first join: the waits -- deliver and join the arms one by one, and a late arm that joined an -- earlier one's writes would be ordered after it, making its reads @@ -111,12 +155,14 @@ local function fanout(worker, collection) fanout = true, }) if not ok then + restore() cancel_live() fail(result) end slot_of[result] = index live[#live + 1] = result end + restore() while #live > 0 do local ok, task, arm_ok, result = yield({ op = "join_any", tasks = live }) if not ok then diff --git a/crates/runvil-internal/lua/src/alias.rs b/crates/runvil-internal/lua/src/alias.rs index 5b385c960..2abcc5947 100644 --- a/crates/runvil-internal/lua/src/alias.rs +++ b/crates/runvil-internal/lua/src/alias.rs @@ -32,12 +32,6 @@ impl Alias { ))) } } - - /// Returns the alias text. - #[must_use] - pub(crate) fn as_str(&self) -> &str { - &self.0 - } } impl From for String { @@ -61,7 +55,7 @@ mod tests { &"z".repeat(64), ] { let parsed = Alias::parse(alias.to_owned()).expect("the alias validates"); - assert_eq!(parsed.as_str(), alias, "{alias:?} keeps its text"); + assert_eq!(String::from(parsed), alias, "{alias:?} keeps its text"); } } diff --git a/crates/runvil-internal/lua/src/bindings-tests.rs b/crates/runvil-internal/lua/src/bindings-tests.rs index ae62c8140..d820d45ca 100644 --- a/crates/runvil-internal/lua/src/bindings-tests.rs +++ b/crates/runvil-internal/lua/src/bindings-tests.rs @@ -40,6 +40,24 @@ fn a_plain_tool_info_binds_with_plain_output() { assert_eq!(binding.output_kind(), ToolOutputKind::Plain); } +#[test] +fn a_new_tool_info_binds_callable_by_a_script() { + assert!( + !ToolBinding::from_info("fetch_alias", &tool_info(false)).model_only_call(), + "a tool info the Pack did not mark stays callable from a script" + ); +} + +#[test] +fn a_model_only_tool_info_binds_with_the_flag_set() { + let info = tool_info(false).model_only_call(true); + let binding = ToolBinding::from_info("fetch_alias", &info); + assert!( + binding.model_only_call(), + "the binding carries the catalog's refusal" + ); +} + #[test] fn the_test_seam_overrides_only_the_description() { let info = tool_info(true); diff --git a/crates/runvil-internal/lua/src/bindings.rs b/crates/runvil-internal/lua/src/bindings.rs index 429afd786..2507d7107 100644 --- a/crates/runvil-internal/lua/src/bindings.rs +++ b/crates/runvil-internal/lua/src/bindings.rs @@ -52,6 +52,9 @@ pub struct ToolBinding { /// Whether the tool's output lists tools, which a script's call /// receives as plain text. describes_tools: bool, + /// Whether only a model may call the tool, so a script's `tools.call` + /// refuses it. + model_only_call: bool, /// Whether a stop leaves a call to the tool in flight. survives_stop: bool, } @@ -74,6 +77,7 @@ impl ToolBinding { ToolOutputKind::Plain }, describes_tools: info.describes_tools, + model_only_call: info.model_only_call, survives_stop: info.survives_stop, } } @@ -115,6 +119,13 @@ impl ToolBinding { self.describes_tools } + /// Returns whether only a model may call the tool, so a script's + /// `tools.call` refuses it. + #[must_use] + pub fn model_only_call(&self) -> bool { + self.model_only_call + } + /// Returns whether a stop leaves a call to the tool in flight. #[must_use] pub fn survives_stop(&self) -> bool { diff --git a/crates/runvil-internal/lua/src/coro-tests.rs b/crates/runvil-internal/lua/src/coro-tests.rs index c4fc52306..404423ce7 100644 --- a/crates/runvil-internal/lua/src/coro-tests.rs +++ b/crates/runvil-internal/lua/src/coro-tests.rs @@ -2,9 +2,16 @@ //! that the one shim install puts `models.loop` and `models.compact` on //! `models`. +use std::sync::{Arc, Mutex}; + +use mlua::{Lua, Value}; +use runvil_model_client::model::{ModelBinding, ModelId, ModelInvocation}; + use super::{ FANOUT_CHUNK_NAME, FANOUT_SOURCE, SHIM_CHUNK_NAME, SHIM_SOURCE, TASKS_CHUNK_NAME, TASKS_SOURCE, + selection_captures, }; +use crate::models::LuaModelHandle; use crate::test_vm::TestVm; use crate::tests::assert_chunk_name_resolves; @@ -34,3 +41,74 @@ fn the_yield_shim_install_puts_models_loop_and_models_compact_on_models() { "the coroutine shim install alone leaves both shims on `models`" ); } + +/// A binding labelled `label`, over the model `model`. +fn binding(label: &str, model: &str) -> ModelBinding { + ModelBinding::new( + label, + "A general model for tests", + ModelId::gateway(model).expect("a static model id is valid"), + ModelInvocation::new(), + std::num::NonZeroU32::new(8192).expect("8192 is non-zero"), + ) +} + +/// The label of the handle `value` holds, when it holds one. +fn handle_label(value: &Value) -> Option { + match value { + Value::UserData(userdata) => userdata + .borrow::() + .ok() + .map(|handle| handle.binding().alias().to_owned()), + _ => None, + } +} + +#[test] +fn get_selection_returns_the_current_handle_or_nil() { + let lua = Lua::new(); + let selection = Arc::new(Mutex::new(None)); + let (get, _) = selection_captures(&lua, &selection).expect("the captures build"); + + let empty: Value = get.call(()).expect("a read of an empty selection succeeds"); + assert!(matches!(empty, Value::Nil), "no model reads as nil"); + + *selection.lock().expect("the selection is not poisoned") = Some(binding("strong", "m2")); + let current: Value = get.call(()).expect("a read of a held selection succeeds"); + assert_eq!(handle_label(¤t).as_deref(), Some("strong")); +} + +#[test] +fn set_selection_stores_a_handle_clears_on_nil_and_refuses_anything_else() { + let lua = Lua::new(); + let selection = Arc::new(Mutex::new(Some(binding("default", "m1")))); + let (_, set) = selection_captures(&lua, &selection).expect("the captures build"); + let label = || { + selection + .lock() + .expect("the selection is not poisoned") + .as_ref() + .map(|held| held.alias().to_owned()) + }; + + let handle = lua + .create_userdata(LuaModelHandle::from_binding(binding("fast", "m3"))) + .expect("a handle builds"); + set.call::<()>(handle).expect("a handle is stored"); + assert_eq!(label().as_deref(), Some("fast")); + + set.call::<()>(Value::Nil) + .expect("nil clears the selection"); + assert_eq!(label(), None); + + let error = set + .call::<()>(5) + .expect_err("a number is neither a handle nor nil"); + assert!( + error + .to_string() + .contains("set_selection takes a model handle or nil, got number"), + "the refusal names the Lua type: {error}" + ); + assert_eq!(label(), None, "a refused write leaves the selection alone"); +} diff --git a/crates/runvil-internal/lua/src/coro.rs b/crates/runvil-internal/lua/src/coro.rs index 2b20e2e47..28024067a 100644 --- a/crates/runvil-internal/lua/src/coro.rs +++ b/crates/runvil-internal/lua/src/coro.rs @@ -25,15 +25,17 @@ mod failure; use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; -use std::sync::{Arc, LazyLock}; +use std::sync::{Arc, LazyLock, Mutex}; use mlua::{Function, Lua, StdLib, Table, Value}; +use runvil_model_client::model::ModelBinding; use crate::engine_globals::route_store_to_shims; use crate::error::SharedSource; use crate::error::{Error, Result}; use crate::error_value::{install_error_value, install_normalize_failure}; use crate::hardening::CancelHook; +use crate::models::{LuaModelHandle, selection_poisoned}; use crate::program::LuaProgram; use crate::var::var_snapshot_table; @@ -65,10 +67,6 @@ const FANOUT_CHUNK_NAME: &str = "@crates/runvil-internal/lua/src/__impl_fanout.l /// a captured cap, bounds how many arms run at once. const FANOUT_SOURCE: &str = include_str!("__impl_fanout.lua"); -/// The registry key for the shim's model-handle methods, `infer` and -/// `loop`, stashed by the prelude install for [`handle_method`]. -const HANDLE_METHODS_REGISTRY: &str = "runvil.impl_coro.handle_methods"; - /// The registry key for the shim's model-issued `tool_call` form, which /// test builds stash for `install_model_tool_call_shim` to install as /// `tools.call_as_model`, so a fixture section can drive the driver's @@ -127,8 +125,9 @@ static FANOUT_PROGRAM: LazyLock> = /// over the same captures plus the collection enumerator and the item /// renderer, and installed on the `tasks` table as `tasks.fanout`. The /// `models.loop` and `models.compact` shims are installed on the `models` -/// table, and the chunk's `handle_methods` table, a model handle's `infer` -/// and `loop`, is stashed in the registry for [`handle_method`]. +/// table. +/// `selection`, the chain's current model, is what the `fanout` chunk's +/// `get_selection` and `set_selection` captures read and write. /// `max_tool_iterations`, the run's resolved round cap, the `compactors` /// table, and `loop_running`, the VM's flag that refuses a loop started /// while one runs, go to [`loop_begin`](crate::models_loop::loop_begin), @@ -147,12 +146,16 @@ static FANOUT_PROGRAM: LazyLock> = /// two captures, `enter_adhoc_handler` and `leave_adhoc_handler`, count up /// and down on `adhoc_handler_depth`, the counter the VM's `jump` reads, so /// `jump` refuses while an adhoc tool's handler runs, whatever reference +/// the two captures, `enter_adhoc_handler` and `leave_adhoc_handler`, count up +/// and down on `adhoc_handler_depth`, the counter the VM's `jump` reads, so +/// `jump` refuses while an adhoc tool's handler runs, whatever reference /// the handler calls it through. Then `cancel_requested` reads the /// cancel flag of `cancel_hook`, so the chunk's protected calls /// raise a failure caught under cancellation again instead of returning /// it, and the hook's abort always reaches the block guard. Next, /// `is_model_handle` reports whether a value is a model handle, so the -/// `infer` entries check their receiver before any yield, and the last +/// `infer` and `loop` entries read a leading handle before any yield, and +/// the last /// two, `loop_begin` and `compact_begin`, start each loop and compaction /// call for the trampoline. /// @@ -162,6 +165,7 @@ static FANOUT_PROGRAM: LazyLock> = pub(crate) fn install_shim_prelude( lua: &Lua, max_tool_iterations: usize, + selection: &Arc>>, adhoc_handler_depth: &Arc, loop_running: &Arc, cancel_hook: &CancelHook, @@ -210,7 +214,7 @@ pub(crate) fn install_shim_prelude( enter_adhoc_handler, leave_adhoc_handler, cancel_requested, - is_model_handle, + is_model_handle.clone(), loop_begin, compact_begin, )) @@ -232,7 +236,14 @@ pub(crate) fn install_shim_prelude( .call((yield_fn.clone(), var_snapshot.clone(), helpers.clone())) .map_err(Error::lua)?; globals.raw_set("tasks", &tasks).map_err(Error::lua)?; - let fanout = fanout_shim(lua, yield_fn, var_snapshot, helpers)?; + let fanout = fanout_shim( + lua, + yield_fn, + var_snapshot, + helpers, + selection, + is_model_handle, + )?; tasks.raw_set("fanout", fanout).map_err(Error::lua)?; stash_shims(lua, &shims)?; for name in ["loop", "compact"] { @@ -246,13 +257,17 @@ pub(crate) fn install_shim_prelude( } /// Runs the `fanout` chunk over the prelude's `yield` and `var_snapshot` -/// captures and failure `helpers`, plus the collection enumerator and the -/// item renderer, and returns the shim installed as `tasks.fanout`. +/// captures and failure `helpers`, plus the collection enumerator, the item +/// renderer, the two captures over the chain's current model `selection`, +/// and the prelude's `is_model_handle`, and returns the shim installed as +/// `tasks.fanout`. fn fanout_shim( lua: &Lua, yield_fn: Function, var_snapshot: Function, helpers: Table, + selection: &Arc>>, + is_model_handle: Function, ) -> Result { // The enumerator answers `(members)` or `(nil, message)` so the shim // raises the message as its own call error, exactly as the other @@ -273,6 +288,7 @@ fn fanout_shim( crate::collection::render_item_value(lua, item).map_err(mlua::Error::external) }) .map_err(Error::lua)?; + let (get_selection, set_selection) = selection_captures(lua, selection)?; FANOUT_PROGRAM .as_ref() .map_err(Error::shared)? @@ -283,17 +299,61 @@ fn fanout_shim( helpers, collection_members, render_item, + get_selection, + set_selection, + is_model_handle, )) .map_err(Error::lua) } -/// Stashes the shim chunk's `handle_methods` and `store` under their -/// registry keys, for the installs and the handle getters that read them -/// after the prelude. -fn stash_shims(lua: &Lua, shims: &Table) -> Result<()> { - let handle_methods: Table = shims.raw_get("handle_methods").map_err(Error::lua)?; - lua.set_named_registry_value(HANDLE_METHODS_REGISTRY, handle_methods) +/// Builds the `fanout` chunk's `get_selection()` and `set_selection(value)` +/// captures over the chain's current model `selection`. +/// +/// `get_selection` returns the current model's handle, or nil when the +/// caller supplied no model. `set_selection` stores the binding of a model +/// handle, clears the selection on nil, and refuses any other value. Both +/// are passed to the chunk and are not reachable from author code. +fn selection_captures( + lua: &Lua, + selection: &Arc>>, +) -> Result<(Function, Function)> { + let read = Arc::clone(selection); + let get = lua + .create_function(move |lua, ()| -> mlua::Result { + let held = read.lock().map_err(|_| selection_poisoned())?.clone(); + match held { + Some(binding) => lua + .create_userdata(LuaModelHandle::from_binding(binding)) + .map(Value::UserData), + None => Ok(Value::Nil), + } + }) .map_err(Error::lua)?; + let write = Arc::clone(selection); + let set = lua + .create_function(move |_, value: Value| -> mlua::Result<()> { + let binding = match &value { + Value::Nil => None, + Value::UserData(userdata) if userdata.is::() => { + Some(userdata.borrow::()?.binding().clone()) + } + other => { + return Err(mlua::Error::external(format!( + "set_selection takes a model handle or nil, got {}", + crate::models::lua_type_name(other) + ))); + } + }; + *write.lock().map_err(|_| selection_poisoned())? = binding; + Ok(()) + }) + .map_err(Error::lua)?; + Ok((get, set)) +} + +/// Stashes the shim chunk's `store` under its registry key, for the store +/// install that reads it after the prelude. +fn stash_shims(lua: &Lua, shims: &Table) -> Result<()> { #[cfg(any(test, feature = "test-support"))] { let model_tool_call: Function = shims.raw_get("model_tool_call").map_err(Error::lua)?; @@ -341,16 +401,6 @@ pub(crate) fn block_guard(lua: &Lua) -> Result { lua.named_registry_value(GUARD_REGISTRY).map_err(Error::lua) } -/// The shim's `name` method for a model handle, `infer` or `loop`, from -/// the table `install_shim_prelude` stashed; nil on a VM where the -/// prelude never ran. -pub(crate) fn handle_method(lua: &Lua, name: &str) -> mlua::Result { - match lua.named_registry_value::(HANDLE_METHODS_REGISTRY)? { - Value::Table(methods) => methods.raw_get(name), - _ => Ok(Value::Nil), - } -} - /// Installs the store yield shims onto a VM's `store` table, replacing the /// dispatchers installed with the other Engine globals, and switches those /// dispatchers to the shims too, so a store function a Pack prelude diff --git a/crates/runvil-internal/lua/src/engine_globals.rs b/crates/runvil-internal/lua/src/engine_globals.rs index f88c5ae93..8d8a7b7be 100644 --- a/crates/runvil-internal/lua/src/engine_globals.rs +++ b/crates/runvil-internal/lua/src/engine_globals.rs @@ -121,7 +121,7 @@ const UI_SNAPSHOT_OPTIONS: mlua::serde::SerializeOptions = mlua::serde::Serializ /// Installs `ui()`: each call converts the run's `snapshot` afresh into a /// new table, JSON nulls reading as nil, so author code that mutates one /// result never sees the mutation on the next call. The snapshot is the -/// application state the caller passed to `RunContext::with_ui` at run start; +/// application state the caller passed to `RunConfig::with_ui` at run start; /// a later change takes effect on the next run. A run started without a /// snapshot never installs the global, so `ui` is absent there - not /// stubbed. diff --git a/crates/runvil-internal/lua/src/error-value.rs b/crates/runvil-internal/lua/src/error-value.rs index 497f11f82..ebb5c9606 100644 --- a/crates/runvil-internal/lua/src/error-value.rs +++ b/crates/runvil-internal/lua/src/error-value.rs @@ -64,6 +64,9 @@ pub enum ErrorKind { OutOfScopeTool, /// A script `tools.call` named an alias with no binding in the run. UnboundTool, + /// A script `tools.call` named a catalog tool that only a model may + /// call, because a render shapes its arguments for the model. + ModelOnlyTool, /// A dispatched tool's own failure. Tool, /// A task operation named a task the caller does not own. @@ -97,6 +100,7 @@ impl ErrorKind { ErrorKind::ModelRequired, ErrorKind::OutOfScopeTool, ErrorKind::UnboundTool, + ErrorKind::ModelOnlyTool, ErrorKind::Tool, ErrorKind::TaskNotOwned, ErrorKind::TaskConsumed, @@ -118,6 +122,7 @@ impl ErrorKind { ErrorKind::ModelRequired => "model_required", ErrorKind::OutOfScopeTool => "out_of_scope_tool", ErrorKind::UnboundTool => "unbound_tool", + ErrorKind::ModelOnlyTool => "model_only_tool", ErrorKind::Tool => "tool", ErrorKind::TaskNotOwned => "task_not_owned", ErrorKind::TaskConsumed => "task_consumed", @@ -140,6 +145,7 @@ impl ErrorKind { "model_required" => Some(ErrorKind::ModelRequired), "out_of_scope_tool" => Some(ErrorKind::OutOfScopeTool), "unbound_tool" => Some(ErrorKind::UnboundTool), + "model_only_tool" => Some(ErrorKind::ModelOnlyTool), "tool" => Some(ErrorKind::Tool), "task_not_owned" => Some(ErrorKind::TaskNotOwned), "task_consumed" => Some(ErrorKind::TaskConsumed), diff --git a/crates/runvil-internal/lua/src/error.rs b/crates/runvil-internal/lua/src/error.rs index 8f9d8aff9..b2e4ccc29 100644 --- a/crates/runvil-internal/lua/src/error.rs +++ b/crates/runvil-internal/lua/src/error.rs @@ -1,10 +1,10 @@ //! The crate's internal error type. //! //! [`Error`] mirrors `runvil-engine`'s own internal error type: it -//! is never part of the documented API. The executor's public boundary -//! (`runvil_engine::RunError`) wraps and classifies `runvil-engine`'s -//! internal error type, which maps this one back variant-for-variant through -//! `From`. The internal type is public only so +//! is never part of the documented API. A failed run's public outcome +//! (`runvil_engine::RunOutcome`) carries the kind and message of +//! `runvil-engine`'s internal error type, which maps this one back +//! variant-for-variant through `From`. The internal type is public only so //! `runvil-engine` can perform that mapping verbatim; the facade does //! not re-export it, and it is not marked `#[non_exhaustive]`, so the //! mapping stays total. @@ -16,8 +16,6 @@ use std::borrow::Cow; -use runvil_model_client::model::ModelSetError; - /// A type-erased owned error cause used by the internal error type. type BoxedSource = Box; @@ -277,16 +275,6 @@ pub(crate) fn author_refusal(message: String) -> mlua::Error { mlua::Error::external(Error::Lua(message)) } -/// Maps a poisoned model-set lock onto [`Error::Internal`], the mapping -/// the Engine receives through its `From`: the lock is -/// the run's own mutex failing, which no prompt author can cause, not a -/// model failure. -impl From for Error { - fn from(error: ModelSetError) -> Error { - Error::internal(error.to_string()) - } -} - /// Crate-internal result alias over [`Error`]. pub(crate) type Result = std::result::Result; diff --git a/crates/runvil-internal/lua/src/lib.rs b/crates/runvil-internal/lua/src/lib.rs index 115b0ac0a..a5982e632 100644 --- a/crates/runvil-internal/lua/src/lib.rs +++ b/crates/runvil-internal/lua/src/lib.rs @@ -49,8 +49,8 @@ //! //! `messages` is the section's one conversation: a list the VM creates //! empty at each section entry and serves as a global that no assignment -//! replaces. `models.loop` and a model handle's `loop` run over it, and a -//! compactor rewrites it in place. +//! can rebind. `models.loop` runs over it, and a compactor rewrites it in +//! place. //! //! Most of this crate's public items exist for `runvil-engine`'s //! executor, which drives the VM and the coroutine protocol; the facade @@ -131,7 +131,6 @@ pub use crate::error_value::{ }; pub use crate::globals::{RESERVED_NAMES, Reserved, reserved_name}; pub use crate::messages::MessageList; -pub use crate::models::ModelSelection; pub use crate::prelude::install_preludes; pub use crate::program::LuaProgram; pub use crate::projection::project_messages; @@ -148,5 +147,5 @@ pub use crate::scope::{ pub use crate::section::{Library, LibraryBlock, LibraryKey, LuaBlockResult, SectionText}; pub use crate::vm::{ AdhocTools, CoroStep, InjectFailure, Injected, SectionVm, Uninjected, current_catalog_packs, - current_dynamic_set, current_static_set, resolve_model_binding, + current_dynamic_set, current_static_set, }; diff --git a/crates/runvil-internal/lua/src/models-tests.rs b/crates/runvil-internal/lua/src/models-tests.rs index 26edb4936..5500460d6 100644 --- a/crates/runvil-internal/lua/src/models-tests.rs +++ b/crates/runvil-internal/lua/src/models-tests.rs @@ -1,82 +1,53 @@ -//! Tests for the `models` namespace: `use`, `default`, `get`, and the model runtime selection. - -use std::sync::Arc; +//! Tests for the `models` namespace: `use`, `get`, and the section's current model. use mlua::Value; use runvil_model_client::model::{ModelBinding, ModelId, ModelInvocation, ModelSet, Temperature}; -use super::{ModelSelection, UseOptions, option_check_order}; -use crate::alias::Alias; +use super::option_check_order; use crate::test_vm::{BuiltVm, TestVm}; -/// The validated alias `text` names. -#[track_caller] -fn alias(text: &str) -> Alias { - Alias::parse(text.to_owned()).expect("a valid alias") -} - -/// The runtime's selected label, if any. -fn used(runtime: &ModelSelection) -> Option<&str> { - runtime.selection().map(|(label, _)| label.as_str()) -} - -#[test] -fn model_runtime_select_allows_reselection() { - // Selections are read at call time, so a later `models.use` replaces the - // earlier one and steers the next model round. - let mut runtime = ModelSelection::new(); - assert!(used(&runtime).is_none()); - runtime.select(alias("writer"), UseOptions::default()); - assert_eq!(used(&runtime), Some("writer")); - runtime.select(alias("other"), UseOptions::default()); - assert_eq!( - used(&runtime), - Some("other"), - "a second select must replace the first" - ); -} +/// The role-name list every label rejection ends with. +const ROLE_LIST: &str = "expected default, strong, small, fast or creative"; -#[test] -fn model_runtime_starts_with_no_selection() { - let runtime = ModelSelection::new(); - assert!(used(&runtime).is_none(), "fresh runtime has no selection"); -} - -/// A bound role for the shared set: `label`, with the keyword set recorded. -fn bound_role(label: &str, capabilities: &[&str]) -> ModelBinding { +/// A bound role for the shared set, under `label`, on the model `model`. +fn bound_role(label: &str, model: &str) -> ModelBinding { ModelBinding::new( label, "A general model for tests", - ModelId::gateway("m1").expect("a static model id is valid"), + ModelId::gateway(model).expect("a static model id is valid"), ModelInvocation::new(), std::num::NonZeroU32::new(8192).expect("8192 is non-zero"), ) - .with_capabilities(capabilities.iter().map(|word| (*word).to_owned()).collect()) } /// A fresh VM with the `models` table installed over a shared set holding -/// the `writer` and `critic` roles. +/// the `default` and `strong` roles. #[track_caller] fn models_vm() -> BuiltVm { let mut models = ModelSet::default(); - models.push(bound_role("writer", &["frontier", "thinking"])); - models.push(bound_role("critic", &["fast"])); + models.push(bound_role("default", "m1")); + models.push(bound_role("strong", "m2")); TestVm::new().models(models).build() } +/// The label of the section's current model, if it has one. +#[track_caller] +fn current_label(vm: &BuiltVm) -> Option { + vm.current_model() + .expect("the current model is readable") + .map(|binding| binding.alias().to_owned()) +} + #[test] -fn the_models_namespace_has_no_bind() { +fn the_models_namespace_offers_use_and_get_and_leaves_infer_to_the_shims() { // `models.infer` suspends, so only the coroutine shim layer defines it. let vm = models_vm(); - let (bind_is_nil, has_use, has_default, has_get, infer_is_nil): (bool, bool, bool, bool, bool) = - vm.eval( - "return models.bind == nil, \ - type(models.use) == 'function', \ - type(models.default) == 'function', \ - type(models.get) == 'function', \ - models.infer == nil", - ); - assert!(bind_is_nil && has_use && has_default && has_get); + let (has_use, has_get, infer_is_nil): (bool, bool, bool) = vm.eval( + "return type(models.use) == 'function', \ + type(models.get) == 'function', \ + models.infer == nil", + ); + assert!(has_use && has_get); assert!( infer_is_nil, "install_models alone must not define models.infer" @@ -84,39 +55,96 @@ fn the_models_namespace_has_no_bind() { } #[test] -fn a_handle_reads_nil_methods_where_the_shim_prelude_never_ran() { +fn a_handle_exposes_fields_and_no_methods() { + // A handle is an inspectable value: the fields read, and a name it does + // not have raises, so `infer` and `loop` are absent however the VM is + // built. let vm = models_vm(); - let (infer_is_nil, loop_is_nil): (bool, bool) = - vm.eval("local h = models.get('writer'); return h.infer == nil, h.loop == nil"); + let (name, infer_failed, loop_failed): (String, bool, bool) = vm.eval( + "local h = models.get('default') + local _, infer_err = pcall(function() return h.infer end) + local _, loop_err = pcall(function() return h.loop end) + return h.name, infer_err ~= nil, loop_err ~= nil", + ); + assert_eq!(name, "default"); assert!( - infer_is_nil && loop_is_nil, - "with no shim prelude a handle's infer and loop must read nil" + infer_failed && loop_failed, + "a handle has no infer and no loop" ); } #[test] -fn models_use_selects_a_bound_role_by_label() { +fn a_new_section_vm_starts_on_the_default_role() { + assert_eq!( + current_label(&models_vm()).as_deref(), + Some("default"), + "a section starts on the default role's binding" + ); +} + +#[test] +fn a_section_vm_with_no_models_has_no_current_model() { + let vm = TestVm::new().build(); + assert_eq!( + current_label(&vm), + None, + "an empty set leaves the section with no model" + ); +} + +#[test] +fn models_use_selects_a_role_by_label() { let vm = models_vm(); - let handle: String = vm.eval("local h = models.use('writer'); return h.label .. '|' .. h.name"); - assert_eq!(handle, "writer|writer"); + let handle: String = vm.eval("local h = models.use('strong'); return h.label .. '|' .. h.name"); + assert_eq!(handle, "strong|strong"); + assert_eq!(current_label(&vm).as_deref(), Some("strong")); +} + +#[test] +fn models_use_rejects_a_label_outside_the_five_roles() { + let vm = models_vm(); + let error = vm.eval_err("models.use('reviewer')"); + assert!( + error.to_string().contains(&format!( + "models.use label \"reviewer\" is not a model role: {ROLE_LIST}" + )), + "the rejection names the label and the five roles: {error}" + ); assert_eq!( - used( - &vm.model_selection() - .lock() - .expect("the model runtime mutex is not poisoned") - ), - Some("writer") + current_label(&vm).as_deref(), + Some("default"), + "a rejected call must not select" ); } #[test] -fn models_use_rejects_an_unbound_label() { - let error = models_vm().eval_err("models.use('ghost')"); +fn models_get_rejects_a_label_outside_the_five_roles() { + let error = models_vm().eval_err("models.get('reviewer')"); assert!( - error - .to_string() - .contains("models.use label \"ghost\" is not a bound model role"), - "the rejection names the label: {error}" + error.to_string().contains(&format!( + "models.get label \"reviewer\" is not a model role: {ROLE_LIST}" + )), + "the rejection names the label and the five roles: {error}" + ); +} + +#[test] +fn a_role_with_no_model_fails_with_the_caller_supplied_none_message() { + for chunk in ["models.use('default')", "models.get('default')"] { + let error = TestVm::new().build().eval_err(chunk); + assert!( + error.to_string().contains("the caller supplied no model"), + "{chunk}: the failure says the caller supplied no model: {error}" + ); + } +} + +#[test] +fn a_role_the_set_lacks_fails_with_the_caller_supplied_none_message() { + let error = models_vm().eval_err("models.use('fast')"); + assert!( + error.to_string().contains("the caller supplied no model"), + "a valid role with no binding says the caller supplied no model: {error}" ); } @@ -124,25 +152,18 @@ fn models_use_rejects_an_unbound_label() { fn models_use_options_reach_the_returned_handle() { let vm = models_vm(); let (temperature, max_tokens): (f64, u32) = vm.eval( - "local h = models.use('writer', { temperature = 0.3, max_tokens = 256 }); \ + "local h = models.use('default', { temperature = 0.3, max_tokens = 256 }); \ return h.temperature, h.max_tokens", ); assert!((temperature - 0.3).abs() < f64::EPSILON); assert_eq!(max_tokens, 256); - assert_eq!( - used( - &vm.model_selection() - .lock() - .expect("the model runtime mutex is not poisoned") - ), - Some("writer") - ); + assert_eq!(current_label(&vm).as_deref(), Some("default")); } #[test] fn models_use_accepts_an_integer_temperature_and_leaves_omitted_fields_nil() { let (temperature, max_tokens_is_nil): (f64, bool) = models_vm().eval( - "local h = models.use('writer', { temperature = 0 }); return h.temperature, h.max_tokens == nil", + "local h = models.use('default', { temperature = 0 }); return h.temperature, h.max_tokens == nil", ); assert!(temperature.abs() < f64::EPSILON); assert!( @@ -155,51 +176,51 @@ fn models_use_accepts_an_integer_temperature_and_leaves_omitted_fields_nil() { fn models_use_rejects_invalid_options_without_selecting() { let cases = [ ( - "models.use('writer', { temperature = 2.5 })", + "models.use('strong', { temperature = 2.5 })", "models.use option temperature 2.5 is outside the supported range [0.0, 2.0]", ), ( - "models.use('writer', { temperature = -0.1 })", + "models.use('strong', { temperature = -0.1 })", "models.use option temperature -0.1 is outside the supported range [0.0, 2.0]", ), ( - "models.use('writer', { temperature = 0/0 })", + "models.use('strong', { temperature = 0/0 })", "models.use option temperature must be finite, got NaN", ), ( - "models.use('writer', { temperature = 'hot' })", + "models.use('strong', { temperature = 'hot' })", "models.use option temperature must be a number, got string", ), ( - "models.use('writer', { max_tokens = 0 })", + "models.use('strong', { max_tokens = 0 })", "models.use option max_tokens must be an integer in [1, 4294967295], got 0", ), ( - "models.use('writer', { max_tokens = -1 })", + "models.use('strong', { max_tokens = -1 })", "models.use option max_tokens must be an integer in [1, 4294967295], got -1", ), ( - "models.use('writer', { max_tokens = 1.5 })", + "models.use('strong', { max_tokens = 1.5 })", "models.use option max_tokens must be an integer in [1, 4294967295], got 1.5", ), ( - "models.use('writer', { max_tokens = 'many' })", + "models.use('strong', { max_tokens = 'many' })", "models.use option max_tokens must be an integer in [1, 4294967295], got string", ), ( - "models.use('writer', { top_p = 0.9 })", + "models.use('strong', { top_p = 0.9 })", "models.use option \"top_p\" is unknown: expected temperature or max_tokens", ), ( - "models.use('writer', { 0.5 })", + "models.use('strong', { 0.5 })", "models.use option names must be strings, got integer", ), ( - "models.use('writer', 'fast')", + "models.use('strong', 'fast')", "models.use options must be a table, got string", ), ( - "models.use('writer', {}, 1)", + "models.use('strong', {}, 1)", "models.use takes at most 2 arguments, got 3", ), ]; @@ -210,13 +231,9 @@ fn models_use_rejects_invalid_options_without_selecting() { error.to_string().contains(expected), "{chunk}: expected {expected:?} in {error}" ); - assert!( - used( - &vm.model_selection() - .lock() - .expect("the model runtime mutex is not poisoned") - ) - .is_none(), + assert_eq!( + current_label(&vm).as_deref(), + Some("default"), "{chunk}: a rejected call must not select" ); } @@ -250,7 +267,7 @@ fn models_use_reports_the_first_bad_option_in_key_order_on_every_state() { expected.contains(check_order[0]), "{options}: the expected rejection {expected:?} names the first key in check order" ); - let chunk = format!("models.use('writer', {options})"); + let chunk = format!("models.use('strong', {options})"); for state in 0..32 { let vm = models_vm(); let table: mlua::Table = vm.eval(&format!("return {options}")); @@ -279,11 +296,12 @@ fn models_use_reports_the_first_bad_option_in_key_order_on_every_state() { } } -/// The section's effective binding's `(temperature, max_tokens)`, read the -/// way the Engine's Chat-effect sites read it. +/// The section's current model's `(temperature, max_tokens)`, read the way +/// the Engine's Chat-effect sites read it. #[track_caller] fn effective_sampling(vm: &BuiltVm) -> (Option, Option) { - let binding = crate::resolve_model_binding(vm.models.as_ref(), vm.model_selection()) + let binding = vm + .current_model() .expect("the model state is readable") .expect("a selection resolves"); let invocation = binding.invocation(); @@ -296,11 +314,11 @@ fn effective_sampling(vm: &BuiltVm) -> (Option, Option) { #[test] fn the_selection_carries_its_options_until_a_later_models_use_replaces_them() { let vm = models_vm(); - vm.eval::<()>("models.use('writer', { temperature = 0.5, max_tokens = 64 })"); + vm.eval::<()>("models.use('default', { temperature = 0.5, max_tokens = 64 })"); assert_eq!(effective_sampling(&vm), (Some(0.5), Some(64))); - let (temperature_is_nil, max_tokens_is_nil): (bool, bool) = - vm.eval("local h = models.use('writer'); return h.temperature == nil, h.max_tokens == nil"); + let (temperature_is_nil, max_tokens_is_nil): (bool, bool) = vm + .eval("local h = models.use('default'); return h.temperature == nil, h.max_tokens == nil"); assert!(temperature_is_nil && max_tokens_is_nil); assert_eq!( effective_sampling(&vm), @@ -310,72 +328,109 @@ fn the_selection_carries_its_options_until_a_later_models_use_replaces_them() { } #[test] -fn a_models_get_handle_for_the_selected_label_keeps_the_role_defaults() { +fn a_models_get_handle_for_the_selected_role_keeps_the_role_defaults() { let (temperature_is_nil, max_tokens_is_nil): (bool, bool) = models_vm().eval( - "models.use('writer', { temperature = 0.5, max_tokens = 64 }); \ - local h = models.get('writer'); \ + "models.use('default', { temperature = 0.5, max_tokens = 64 }); \ + local h = models.get('default'); \ return h.temperature == nil, h.max_tokens == nil", ); assert!(temperature_is_nil && max_tokens_is_nil); } #[test] -fn models_default_takes_a_label_and_parks_the_prompt_wide_default() { - let vm = models_vm(); - vm.eval::<()>("models.default('writer')"); +fn the_handle_exposes_label_model_id_description_context_thinking_and_provider() { + let mut models = ModelSet::default(); + models.push( + bound_role("default", "m1") + .with_invocation(ModelInvocation::new().with_thinking(Some(true))) + .with_provider("acme"), + ); + let inspected: String = TestVm::new().models(models).build().eval( + "local h = models.get('default'); \ + return h.label .. '|' .. h.model_id .. '|' .. h.description .. '|' .. h.context \ + .. '|' .. tostring(h.thinking) .. '|' .. h.provider", + ); assert_eq!( - vm.models - .lock() - .expect("the model set mutex is not poisoned") - .default_alias(), - Some("writer") + inspected, + "default|m1|A general model for tests|8192|true|acme" ); } #[test] -fn models_default_is_idempotent_for_the_same_label_and_refuses_a_change() { - let vm = models_vm(); - // A library every section requires re-runs in each, so re-naming the - // same default must be a no-op. - vm.eval::<()>("models.default('writer'); models.default('writer')"); - assert_eq!( - vm.models - .lock() - .expect("the model set mutex is not poisoned") - .default_alias(), - Some("writer") - ); - let error = vm.eval_err("models.default('critic')"); +fn models_current_returns_the_default_handle_before_any_use() { + let (label, model_id): (String, String) = + models_vm().eval("local h = models.current(); return h.label, h.model_id"); + assert_eq!((label.as_str(), model_id.as_str()), ("default", "m1")); +} + +#[test] +fn models_current_follows_models_use() { + let (label, model_id): (String, String) = models_vm() + .eval("models.use('strong'); local h = models.current(); return h.label, h.model_id"); + assert_eq!((label.as_str(), model_id.as_str()), ("strong", "m2")); +} + +#[test] +fn models_current_with_no_models_fails_with_the_caller_supplied_none_message() { + let error = TestVm::new().build().eval_err("models.current()"); assert!( - error - .to_string() - .contains("models.default is already \"writer\""), - "the refusal names the parked default: {error}" + error.to_string().contains("the caller supplied no model"), + "the failure says the caller supplied no model: {error}" ); } #[test] -fn models_default_rejects_an_unbound_label() { - let error = models_vm().eval_err("models.default('ghost')"); +fn models_use_of_a_saved_handle_puts_the_section_back_on_it() { + let vm = models_vm(); + vm.eval::<()>("local prev = models.current(); models.use('strong'); models.use(prev)"); + assert_eq!(current_label(&vm).as_deref(), Some("default")); +} + +#[test] +fn models_use_of_a_handle_returns_that_handle() { + let (label, model_id): (String, String) = + models_vm().eval("local h = models.use(models.get('strong')); return h.label, h.model_id"); + assert_eq!((label.as_str(), model_id.as_str()), ("strong", "m2")); +} + +#[test] +fn models_use_applies_its_options_on_top_of_a_handle() { + let vm = models_vm(); + vm.eval::<()>("local prev = models.current(); models.use(prev, { temperature = 0.2 })"); + assert_eq!(current_label(&vm).as_deref(), Some("default")); + assert_eq!(effective_sampling(&vm), (Some(0.2), None)); +} + +#[test] +fn models_use_accepts_a_raw_id_handle_from_models_get() { + let vm = TestVm::new().raw_model_ids().build(); + vm.eval::<()>("models.use(models.get('vendor/raw-model'))"); + let binding = vm + .current_model() + .expect("the current model is readable") + .expect("a selection resolves"); + assert_eq!(binding.alias(), "vendor/raw-model"); + assert_eq!(binding.id().name(), "vendor/raw-model"); +} + +#[test] +fn models_use_rejects_an_argument_that_is_neither_a_role_nor_a_handle() { + let vm = models_vm(); + let error = vm.eval_err("models.use(5)"); assert!( error .to_string() - .contains("models.default label \"ghost\" is not a bound model role"), - "the rejection names the label: {error}" + .contains("models.use takes a model role name or a model handle, got number"), + "the rejection names the argument's Lua type: {error}" ); -} - -#[test] -fn the_handle_exposes_label_and_the_full_keyword_set() { - let inspected: String = models_vm().eval( - "local h = models.get('writer'); \ - return h.label .. '|' .. h.model_id .. '|' .. table.concat(h.capabilities, ',')", + assert_eq!( + current_label(&vm).as_deref(), + Some("default"), + "a rejected call must not select" ); - assert_eq!(inspected, "writer|m1|frontier,thinking"); } - #[test] -fn models_get_resolves_an_undeclared_alias_as_a_raw_gateway_id_only_when_permitted() { +fn models_get_resolves_a_label_outside_the_roles_as_a_raw_gateway_id_only_when_permitted() { let resolved: String = TestVm::new().raw_model_ids().build().eval( "local h = models.get('qwen/qwen3-8b'); return h.name .. '|' .. h.model_id .. '|' .. h.context", ); @@ -386,8 +441,8 @@ fn models_get_resolves_an_undeclared_alias_as_a_raw_gateway_id_only_when_permitt let error = TestVm::new().build().eval_err("models.get('ghost')"); assert!( - error.to_string().contains("is not a bound model role"), - "the strict path names the bound-role rule: {error}" + error.to_string().contains("is not a model role"), + "the strict path names the role rule: {error}" ); } @@ -399,28 +454,25 @@ fn the_raw_id_fallback_still_validates_the_gateway_id() { .eval_err("models.get('bad\\nid')"); assert!( error.to_string().contains("is invalid"), - "the raw path validates as a gateway id, not as an alias: {error}" + "the raw path validates as a gateway id, not as a role label: {error}" ); } #[test] -fn a_poisoned_model_set_reaches_the_caller_as_an_internal_error() { +fn a_poisoned_selection_reaches_the_caller_as_an_internal_error() { let vm = models_vm(); - vm.model_selection() - .lock() - .expect("the model runtime mutex is not poisoned") - .select(alias("writer"), UseOptions::default()); - let poisoner = Arc::clone(&vm.models); + let poisoner = std::sync::Arc::clone(&vm.selection); let _ = std::thread::spawn(move || { let _guard = poisoner.lock(); - panic!("poison the model set"); + panic!("poison the model selection"); }) .join(); - let error = crate::resolve_model_binding(vm.models.as_ref(), vm.model_selection()) - .expect_err("a poisoned model set cannot resolve a binding"); + let error = vm + .current_model() + .expect_err("a poisoned selection cannot be read"); assert!( - matches!(&error, crate::Error::Internal { message, .. } if message == "model set mutex was poisoned"), + matches!(&error, crate::Error::Internal { message, .. } if message == "model selection lock was poisoned"), "the lock failure is Engine misuse, neither an authoring nor a model failure: {error:?}" ); } diff --git a/crates/runvil-internal/lua/src/models-userdata.rs b/crates/runvil-internal/lua/src/models-userdata.rs index 5b6cbf190..fc7ebf999 100644 --- a/crates/runvil-internal/lua/src/models-userdata.rs +++ b/crates/runvil-internal/lua/src/models-userdata.rs @@ -1,11 +1,10 @@ -//! Inspectable Lua userdata returned by `models.use` / `models.default` / -//! `models.get`. +//! Inspectable Lua userdata returned by `models.use`, `models.get`, and +//! `models.current`. //! -//! The userdata exposes a frozen [`ModelBinding`]'s fields to Lua, and it -//! carries `infer` and `loop`, run as `h:infer(prompt)` and -//! `h:loop(compactor?)` on the handle's binding. Those two are -//! the shim's Lua functions, read through field getters, because each may -//! suspend and a Rust method cannot. +//! The userdata is a pure value: it exposes a frozen [`ModelBinding`]'s +//! fields to Lua and nothing else. It has no methods, so there is exactly +//! one way to run a model: the `models` namespace functions, which take the +//! handle as their leading argument. use mlua::{UserData, UserDataFields, Value}; @@ -39,12 +38,6 @@ impl LuaModelHandle { self.binding.alias() } - /// Returns the bound role's full keyword set. - #[must_use] - fn capabilities(&self) -> &[String] { - self.binding.capabilities() - } - /// Returns the name of the bound model, without the server namespace /// its id also holds. #[must_use] @@ -52,7 +45,7 @@ impl LuaModelHandle { self.binding.id().name() } - /// Returns the capability description of the bound role. + /// Returns the description of the bound model. #[must_use] fn description(&self) -> &str { self.binding.description() @@ -70,13 +63,13 @@ impl LuaModelHandle { self.binding.provider().map(str::to_owned) } - /// Returns the frozen thinking switch, when the role declared one. + /// Returns the frozen thinking switch, when the role set one. #[must_use] fn thinking(&self) -> Option { self.binding.invocation().thinking } - /// Returns the frozen sampling temperature, when the role declared one. + /// Returns the frozen sampling temperature, when one was set. #[must_use] fn temperature(&self) -> Option { self.binding @@ -85,7 +78,7 @@ impl LuaModelHandle { .map(runvil_model_client::model::Temperature::get) } - /// Returns the frozen max generation tokens, when the role declared one. + /// Returns the frozen max generation tokens, when one was set. #[must_use] fn max_tokens(&self) -> Option { self.binding @@ -99,9 +92,6 @@ impl UserData for LuaModelHandle { fn add_fields>(fields: &mut F) { fields.add_field_method_get("name", |_, this| Ok(this.name().to_owned())); fields.add_field_method_get("label", |_, this| Ok(this.name().to_owned())); - fields.add_field_method_get("capabilities", |lua, this| { - lua.create_sequence_from(this.capabilities().to_vec()) - }); fields.add_field_method_get("model_id", |_, this| Ok(this.model_id().to_owned())); fields.add_field_method_get("description", |_, this| Ok(this.description().to_owned())); fields.add_field_method_get("context", |_, this| Ok(this.context())); @@ -109,8 +99,6 @@ impl UserData for LuaModelHandle { fields.add_field_method_get("thinking", |_, this| Ok(this.thinking())); fields.add_field_method_get("temperature", |_, this| Ok(this.temperature())); fields.add_field_method_get("max_tokens", |_, this| Ok(this.max_tokens())); - fields.add_field_function_get("infer", |lua, _| crate::coro::handle_method(lua, "infer")); - fields.add_field_function_get("loop", |lua, _| crate::coro::handle_method(lua, "loop")); } } diff --git a/crates/runvil-internal/lua/src/models.rs b/crates/runvil-internal/lua/src/models.rs index b89ec7393..fc8f06ee7 100644 --- a/crates/runvil-internal/lua/src/models.rs +++ b/crates/runvil-internal/lua/src/models.rs @@ -1,13 +1,13 @@ -//! The Lua `models` Engine global: `use` / `default` / `get`. +//! The Lua `models` Engine global: `use` / `get` / `current`. //! -//! Binding is frontmatter: the run's roles arrive pre-filled from prepare in -//! the shared [`ModelSet`], and the table selects among them by label. -//! `models.use` records the section's selection with its optional sampling -//! options, `models.default` parks the prompt-wide default, and `models.get` -//! inspects a bound role without selecting it. The suspending -//! `models.infer`, `models.loop`, and `models.compact` are coroutine shims -//! added to this table afterward. - +//! The five roles arrive filled, when the run is built, in the shared +//! [`ModelSet`], and the table selects among them by role name. `models.use` takes a role +//! name or a model handle and stores the binding it selects, with its +//! optional sampling options applied, as the section's current model. +//! `models.get` inspects a role without selecting it, and `models.current` +//! returns the handle for the current model. The suspending `models.infer`, +//! `models.loop`, and `models.compact` are coroutine shims added to this +//! table afterward. use std::num::NonZeroU32; use std::sync::Arc; use std::sync::Mutex; @@ -17,8 +17,8 @@ use mlua::{Lua, MultiValue, Table, Value}; use runvil_model_client::model::{ ModelBinding, ModelId, ModelInvocation, ModelSet, Temperature, TemperatureError, }; +use runvil_types::models::ModelRole; -use crate::alias::Alias; use crate::error::{Error, Result}; #[path = "models-userdata.rs"] @@ -26,13 +26,13 @@ mod userdata; pub(crate) use userdata::{LuaModelHandle, is_handle}; -/// Whether `models.get` resolves an alias no role declares as a raw +/// Whether `models.get` resolves a label outside the five roles as a raw /// gateway catalog model id. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum RawModelIds { - /// An undeclared alias is an error. + /// A label outside the five roles is an error. Refused, - /// An undeclared alias resolves as a raw gateway catalog model id. + /// A label outside the five roles resolves as a raw gateway catalog model id. Resolved, } @@ -44,9 +44,9 @@ const RAW_ID_CONTEXT: NonZeroU32 = match NonZeroU32::new(8192) { None => unreachable!(), }; -/// Builds the raw-model-id fallback's binding: an undeclared `models.get` -/// alias resolved as a raw gateway catalog model id, with no invocation -/// overrides and the fallback context window. +/// Builds the raw-model-id fallback's binding: a `models.get` label outside +/// the five roles, resolved as a raw gateway catalog model id with no +/// invocation overrides and the fallback context window. fn raw_gateway_binding(alias: &str) -> mlua::Result { let id = ModelId::gateway(alias).map_err(|error| { mlua::Error::external(format!("models.get model id {alias:?} is invalid: {error}")) @@ -60,13 +60,55 @@ fn raw_gateway_binding(alias: &str) -> mlua::Result { )) } -/// Locks the run's shared model set, mapping a poisoned lock to the Lua -/// boundary error every Engine function uses. -fn lock_models(set: &Mutex) -> mlua::Result> { - set.lock() - .map_err(|_| mlua::Error::external("model set mutex was poisoned")) +/// The error every selection read or write raises when its lock is poisoned. +pub(crate) fn selection_poisoned() -> mlua::Error { + mlua::Error::external("model selection lock was poisoned") +} + +/// The Lua type name of `value`, as `type(value)` reports it: an integer is a +/// `number`. +pub(crate) fn lua_type_name(value: &Value) -> &'static str { + match value { + Value::Integer(_) | Value::Number(_) => "number", + other => other.type_name(), + } +} + +/// The five role names as a sentence tail: `default, strong, small, fast or +/// creative`. +fn role_list() -> String { + let names: Vec<&str> = ModelRole::ALL.iter().map(|role| role.as_str()).collect(); + match names.split_last() { + Some((last, rest)) => format!("{} or {last}", rest.join(", ")), + None => String::new(), + } +} + +/// The role `label` names, if it names one. +fn find_role(label: &str) -> Option { + ModelRole::ALL + .into_iter() + .find(|role| role.as_str() == label) } +/// The `function` rejection of a label that is not a role, listing the five. +fn not_a_role(function: &str, label: &str) -> mlua::Error { + mlua::Error::external(format!( + "{function} label {label:?} is not a model role: expected {}", + role_list() + )) +} + +/// The binding that fills `role`, or the error saying the caller supplied +/// no model for it. +fn role_binding(set: &ModelSet, role: ModelRole) -> mlua::Result { + set.binding(role.as_str()).cloned().ok_or_else(|| { + mlua::Error::external(format!( + "the caller supplied no model for the {} role", + role.as_str() + )) + }) +} /// The sampling options a `models.use` call sets on its selection. An /// omitted field keeps the role's own value. #[derive(Debug, Clone, Copy, Default, PartialEq)] @@ -195,52 +237,23 @@ fn parse_use_options(options: Value, rest: &MultiValue) -> mlua::Result, -} - -impl ModelSelection { - pub(crate) fn new() -> Self { - ModelSelection { selection: None } - } - - /// Returns the current `models.use` label and its options, if any. - pub(crate) fn selection(&self) -> Option<(&Alias, UseOptions)> { - self.selection - .as_ref() - .map(|(label, options)| (label, *options)) - } - - /// Records a `models.use` selection, replacing any prior label and - /// options: the selection is read at call time, so the latest call - /// steers the next model round. - fn select(&mut self, alias: Alias, options: UseOptions) { - self.selection = Some((alias, options)); - } -} - /// Installs the `models` table into one section VM (H1 included: there is /// one install path for every section). /// -/// The table reads and writes the run's shared [`ModelSet`]: -/// `models.use(label, options?)` records the section's own selection in -/// `runtime`, with an optional `temperature` / `max_tokens` table that -/// applies to rounds on that selection and to the handle it returns, while -/// `models.default(label)` records the prompt-wide default in the shared -/// set - a static prompt-wide fact, conventionally called from H1 but not -/// privileged to it. Re-selecting the same label is a no-op, so a library -/// section that every section loads with `require` may name the default; -/// naming a different label errors. There is no `models.bind`: binding is the -/// frontmatter's, and an unknown label is a hard error. +/// `models.use(role_or_handle, options?)` takes a role name, which it looks up +/// in the shared [`ModelSet`], or a model handle, whose binding it uses as it +/// is. It applies the optional `temperature` / `max_tokens` table on top of +/// that binding, stores the result in `selection` as the section's current +/// model, and returns the handle. `models.get(label)` returns the handle for +/// a role without selecting it, and `models.current()` returns the handle +/// for the current model. A label outside the five roles is an error, and a +/// role the caller gave no model fails saying so. /// /// `raw_ids` is the raw-model-id fallback, resolved whenever the caller -/// passes an application-state snapshot (`RunContext::with_ui`): then -/// `models.get` resolves an undeclared alias as a raw gateway catalog model -/// id, so the built-in chat prompt can run `models.get(ui().selected_model)` -/// without declaring its model. Refused, an undeclared alias is the usual +/// passes an application-state snapshot (`RunConfig::with_ui`): then +/// `models.get` resolves a label outside the five roles as a raw gateway +/// catalog model id, so the built-in chat prompt can run +/// `models.get(ui().selected_model)`. Refused, such a label is the usual /// error. /// /// The coroutine shim layer installs the suspending `models.infer`, @@ -253,100 +266,75 @@ impl ModelSelection { pub(crate) fn install_models( lua: &Lua, globals: &Table, - set: &Arc>, - runtime: &Arc>, + set: &Arc, + selection: &Arc>>, raw_ids: RawModelIds, ) -> Result<()> { let models = lua.create_table().map_err(Error::lua)?; let frozen = Arc::clone(set); - let state = Arc::clone(runtime); + let current = Arc::clone(selection); let use_fn = lua .create_function( move |_, - (label, options, rest): (String, Value, MultiValue)| + (target, options, rest): (Value, Value, MultiValue)| -> mlua::Result { - let label = Alias::parse(label).map_err(mlua::Error::external)?; - let binding = lock_models(&frozen)? - .binding(label.as_str()) - .cloned() - .ok_or_else(|| { - mlua::Error::external(format!( - "models.use label {:?} is not a bound model role", - label.as_str() - )) - })?; + let binding = match &target { + Value::String(text) => { + let label = text.to_string_lossy(); + let role = + find_role(&label).ok_or_else(|| not_a_role("models.use", &label))?; + role_binding(&frozen, role)? + } + Value::UserData(userdata) if userdata.is::() => { + userdata.borrow::()?.binding().clone() + } + other => { + return Err(mlua::Error::external(format!( + "models.use takes a model role name or a model handle, got {}", + lua_type_name(other) + ))); + } + }; let options = parse_use_options(options, &rest)?; - let mut state = state - .lock() - .map_err(|_| mlua::Error::external("model selection lock was poisoned"))?; - state.select(label, options); - Ok(LuaModelHandle::from_binding(options.apply(binding))) + let binding = options.apply(binding); + *current.lock().map_err(|_| selection_poisoned())? = Some(binding.clone()); + Ok(LuaModelHandle::from_binding(binding)) }, ) .map_err(Error::lua)?; models.set("use", use_fn).map_err(Error::lua)?; let frozen = Arc::clone(set); - let default_fn = lua + let get_fn = lua .create_function(move |_, label: String| -> mlua::Result { - let label = Alias::parse(label).map_err(mlua::Error::external)?; - let mut set = lock_models(&frozen)?; - let binding = set - .binding(label.as_str()) - .cloned() - .ok_or_else(|| { - mlua::Error::external(format!( - "models.default label {:?} is not a bound model role", - label.as_str() - )) - })?; - match set.default_alias() { - // Idempotent under `require`: every section that requires - // a library re-runs it, so naming the same default again is - // a no-op. - Some(existing) if existing == label.as_str() => {} - Some(existing) => { - return Err(mlua::Error::external(format!( - "models.default is already {existing:?}: the prompt-wide default cannot change mid-run" - ))); - } - None => set.set_default_alias(label.into()), - } + // The raw-model-id fallback: with raw ids resolved, a label + // outside the five roles resolves as a raw gateway catalog model + // id under the fallback context window. Gateway ids include `/`, + // `.`, and `:`, so the id's own validation is the only gate. + let binding = match (find_role(&label), raw_ids) { + (Some(role), _) => role_binding(&frozen, role)?, + (None, RawModelIds::Resolved) => raw_gateway_binding(&label)?, + (None, RawModelIds::Refused) => return Err(not_a_role("models.get", &label)), + }; Ok(LuaModelHandle::from_binding(binding)) }) .map_err(Error::lua)?; - models.set("default", default_fn).map_err(Error::lua)?; + models.set("get", get_fn).map_err(Error::lua)?; - let frozen = Arc::clone(set); - let get_fn = lua - .create_function(move |_, alias: String| -> mlua::Result { - if let Some(binding) = lock_models(&frozen)?.binding(&alias).cloned() { - return Ok(LuaModelHandle::from_binding(binding)); - } - // The raw-model-id fallback: with raw ids resolved, an - // undeclared alias resolves as a raw gateway catalog model id - // under the fallback context window. The alias grammar does not - // apply - gateway ids include `/`, `.`, and `:` - so the id's own - // validation is the only gate. - match raw_ids { - RawModelIds::Resolved => { - return Ok(LuaModelHandle::from_binding(raw_gateway_binding(&alias)?)); - } - RawModelIds::Refused => {} - } - let alias = Alias::parse(alias).map_err(mlua::Error::external)?; - Err(mlua::Error::external(format!( - "models.get alias {:?} is not a bound model role", - alias.as_str() - ))) + let current = Arc::clone(selection); + let current_fn = lua + .create_function(move |_, ()| -> mlua::Result { + let binding = current.lock().map_err(|_| selection_poisoned())?.clone(); + binding + .map(LuaModelHandle::from_binding) + .ok_or_else(|| mlua::Error::external("the caller supplied no model")) }) .map_err(Error::lua)?; - models.set("get", get_fn).map_err(Error::lua)?; + models.set("current", current_fn).map_err(Error::lua)?; globals.raw_set("models", models).map_err(Error::lua) } - #[cfg(test)] #[path = "models-tests.rs"] mod tests; diff --git a/crates/runvil-internal/lua/src/models_loop.rs b/crates/runvil-internal/lua/src/models_loop.rs index bf123eac5..b53f3e8ff 100644 --- a/crates/runvil-internal/lua/src/models_loop.rs +++ b/crates/runvil-internal/lua/src/models_loop.rs @@ -1,10 +1,12 @@ //! The rules of `models.loop` and `models.compact`, run in Rust behind the //! loop shim. //! -//! `models.loop` and a model handle's `loop` are one-line entries over a +//! `models.loop` is a one-line entry over a //! Lua trampoline in `__impl_coro.lua` over `loop_begin`, which this //! module builds, and the step closure each call returns, which owns that -//! call's machine. `models.compact` is a third entry over the same +//! call's machine. `models.loop` takes an optional leading model handle, +//! which runs every round on that handle's frozen binding, and an optional +//! compactor. `models.compact` is a second entry over the same //! trampoline and `compact_begin`, whose step settles one compactor call //! the way an overflowed round's is settled. A Rust function called from //! Lua cannot yield, and cannot call a Lua function that may yield, so @@ -34,20 +36,10 @@ mod machine; use machine::{Input, Machine, Phase, Then, raised, settle}; -/// `models.loop`'s refusal of a model handle in the compactor's place. -const HANDLE_FIRST: &str = "models.loop takes (compactor?); \ - call handle:loop(compactor?) to run on a model handle"; +/// `models.loop`'s refusal past its two argument shapes. +const LOOP_ARITY: &str = "models.loop takes (compactor?) or (model, compactor?)"; -/// `models.loop`'s refusal past one argument. -const LOOP_ARITY: &str = "models.loop takes (compactor?)"; - -/// A handle's `loop` called without a model handle as its receiver. -const NO_RECEIVER: &str = "call loop on a model handle with a colon: handle:loop(compactor?)"; - -/// A handle's `loop` past its receiver and one argument. -const METHOD_ARITY: &str = "handle:loop takes (compactor?)"; - -/// Either entry called while a loop or a compactor runs in the same +/// `models.loop` called while a loop or a compactor runs in the same /// section. const LOOP_RUNNING: &str = "a loop or a compactor is already running in this section; \ ask a side question with models.infer(prompt)"; @@ -62,10 +54,10 @@ const COMPACT_RUNNING: &str = /// Builds `loop_begin`, the shim chunk's argument before `compact_begin`. /// -/// `loop_begin(entry, ...)` takes `false` for `models.loop` or `true` for -/// a handle's `loop`, then that entry's arguments, and returns the -/// call's step closure followed by the first action's tag and values; -/// or `nil`, `"raise"`, and the argument error. Every call runs over the +/// `loop_begin(...)` takes the call's arguments: an optional leading model +/// handle, then an optional compactor. It returns the call's step closure +/// followed by the first action's tag and values; or `nil`, `"raise"`, and +/// the argument error. Every call runs over the /// section's list, read from the `_G` guard's `messages` slot. The step /// closure, made with `create_function_mut`, owns the call's /// [`machine::Machine`]; it takes the last yield's resume values, or the @@ -100,9 +92,8 @@ pub(crate) fn loop_begin( .map_err(Error::lua) } -/// `loop_begin`'s body: the entry flag; for a handle's `loop`, the -/// receiver then the arity, and for `models.loop`, the handle-first -/// refusal then the arity; the compactor's type; the refusal while +/// `loop_begin`'s body: the leading handle, then the arity; the +/// compactor's type; the refusal while /// `running` or the list's `compacting` flag is set; then the machine over /// the section's list, its step closure, and the first action. `running` /// is set only once nothing here can fail and the first action leaves the @@ -118,31 +109,23 @@ fn begin( running: &Arc, hook: &CancelHook, ) -> mlua::Result { - let mut args = args.into_iter(); - let Some(Value::Boolean(method)) = args.next() else { - return Err(malformed("loop_begin needs its entry flag first")); + let args: Vec = args.into_iter().collect(); + // A leading model handle selects the handle's frozen binding; anything + // else is the compactor of the section-model shape. + let handle = match args.split_first() { + Some((first, _)) if crate::models::is_handle(first) => Some(first.clone()), + _ => None, }; - let args: Vec = args.collect(); - let receiver = args.first().is_some_and(crate::models::is_handle); - let (handle, mut args) = if method { - if !receiver { - return refuse(lua, NO_RECEIVER); - } - if args.len() > 2 { - return refuse(lua, METHOD_ARITY); - } - let mut args = args.into_iter(); - (args.next(), args) + let rest = if handle.is_some() { + &args[1..] } else { - if receiver { - return refuse(lua, HANDLE_FIRST); - } - if args.len() > 1 { - return refuse(lua, LOOP_ARITY); - } - (None, args.into_iter()) + &args[..] }; - let compactor = match args.next().unwrap_or(Value::Nil) { + if rest.len() > 1 { + return refuse(lua, LOOP_ARITY); + } + let compactor = rest.first().cloned().unwrap_or(Value::Nil); + let compactor = match compactor { Value::Nil => compactors.get::("fail")?, function @ Value::Function(_) => function, other => { diff --git a/crates/runvil-internal/lua/src/models_loop/machine.rs b/crates/runvil-internal/lua/src/models_loop/machine.rs index e08466659..a4a948640 100644 --- a/crates/runvil-internal/lua/src/models_loop/machine.rs +++ b/crates/runvil-internal/lua/src/models_loop/machine.rs @@ -56,8 +56,8 @@ pub(super) struct Machine { list: MessageList, /// The same list as the value every `chat` request names. messages: V, - /// The receiver of a handle's `loop`, set on every `chat` request; - /// `None` for `models.loop`. + /// The leading model handle's value, set on every `chat` request; + /// `None` when the call named no handle. handle: Option, /// The compactor argument, or `compactors.fail` as read when the /// call began, which is not type-checked, as in the shim. diff --git a/crates/runvil-internal/lua/src/models_loop/tests.rs b/crates/runvil-internal/lua/src/models_loop/tests.rs index 4977976b3..ea0133010 100644 --- a/crates/runvil-internal/lua/src/models_loop/tests.rs +++ b/crates/runvil-internal/lua/src/models_loop/tests.rs @@ -98,7 +98,7 @@ fn begin_with_flag(lua: &Lua, max_rounds: usize) -> (Function, Arc) } #[test] -fn both_entries_check_their_arguments_in_order_with_the_shims_texts() { +fn the_entry_checks_its_arguments_in_order_with_the_shims_texts() { let lua = Lua::new(); install_section_list(&lua); let (begin, _) = begin_with_flag(&lua, 2); @@ -111,39 +111,43 @@ fn both_entries_check_their_arguments_in_order_with_the_shims_texts() { return type(step) .. '|' .. tag .. '|' .. tostring(value.op or value) end return { - probe(false, h), probe(false, h, nil), probe(false, nil, nil), - probe(false, 5, nil), probe(false, 5), probe(false, null), probe(false), - probe(false, nil), probe(true), probe(true, 5), probe(true, h, nil, nil), - probe(true, h, 1.5, nil), probe(true, h, 1.5), probe(true, h), - probe(true, h, nil), + probe(h, nil, nil), probe(h, nil), probe(h, 1.5), + probe(h, 1.5, nil), probe(false, h), probe(5, h), + probe(false, nil, nil), probe(5, nil), probe(5), probe(null), + probe(), probe(nil), probe(h), probe(h, 5), }", ) .call((begin, handle(&lua), Value::NULL)) .unwrap(); - let handle_first = "nil|raise|models.loop takes (compactor?); call handle:loop(compactor?) \ - to run on a model handle"; - let loop_arity = "nil|raise|models.loop takes (compactor?)"; - let no_receiver = "nil|raise|call loop on a model handle with a colon: handle:loop(compactor?)"; - let method_arity = "nil|raise|handle:loop takes (compactor?)"; + let arity = "nil|raise|models.loop takes (compactor?) or (model, compactor?)"; + let integer = "nil|raise|compactor must be a function, got integer"; + let number = "nil|raise|compactor must be a function, got number"; let started = "function|yield|chat"; assert_eq!( outcomes, [ - handle_first, - handle_first, - loop_arity, - loop_arity, - "nil|raise|compactor must be a function, got integer", - "nil|raise|compactor must be a function, got userdata", + // Three arguments, whichever shape the first one starts. + arity, + // A handle then one argument is the handle shape. started, + // A handle then a non-function compactor. + number, + arity, + // A non-handle first argument takes the section-model shape, so + // it may carry only one argument. + arity, + arity, + arity, + arity, + // That one argument is the compactor, checked after the arity. + integer, + "nil|raise|compactor must be a function, got userdata", + // No arguments, a nil argument, a handle alone, and a handle + // with nil all take a documented shape. started, - no_receiver, - no_receiver, - method_arity, - method_arity, - "nil|raise|compactor must be a function, got number", started, started, + integer, ] ); } @@ -162,7 +166,7 @@ fn every_started_loop_yields_the_sections_one_list() { step(false, 'ended') return request.messages end - return messages_sent(false), messages_sent(true, h)", + return messages_sent(), messages_sent(h)", ) .call((begin, handle(&lua))) .unwrap(); @@ -199,16 +203,16 @@ fn a_running_loop_refuses_another_until_its_step_ends_or_fails() { return type(refused) .. '|' .. tag .. '|' .. tostring(err) end note('idle') - local step = begin(false) + local step = begin() note('started') - note('loop ' .. refusal(false)) - note('method ' .. refusal(true, h)) + note('loop ' .. refusal()) + note('handle ' .. refusal(h)) note(step(false, 'ended')) - local _, exhausted = capped(false) + local _, exhausted = capped() note('capped ' .. exhausted) - local failing = begin(false) + local failing = begin() note('restarted') - note(tostring(pcall(failing, true, 'not a round'))) + note(tostring(pcall(failing, 'not a round'))) return table.concat(seen, ',')", ) .call((begin, capped, is_running, handle(&lua))) @@ -217,7 +221,7 @@ fn a_running_loop_refuses_another_until_its_step_ends_or_fails() { assert_eq!( seen, format!( - "idle=false,started=true,loop {refused},method {refused},raise=false,\ + "idle=false,started=true,loop {refused},handle {refused},raise=false,\ capped raise=false,restarted=true,false=false" ) ); @@ -236,7 +240,7 @@ fn a_step_failing_on_a_full_heap_after_an_overflow_leaves_no_flag_set() { let (begin, running) = begin_with_flag(&lua, 2); let mut judged = false; for headroom in (0..64 * 1024).step_by(8) { - let (step, _, _): (Function, Value, Value) = begin.call(false).unwrap(); + let (step, _, _): (Function, Value, Value) = begin.call(()).unwrap(); let overflowed = ChatResult { outcome: ChatOutcome::Overflow(OverflowReason::Precheck), ..round() diff --git a/crates/runvil-internal/lua/src/protocol/parse.rs b/crates/runvil-internal/lua/src/protocol/parse.rs index 4de16963f..e07332646 100644 --- a/crates/runvil-internal/lua/src/protocol/parse.rs +++ b/crates/runvil-internal/lua/src/protocol/parse.rs @@ -157,10 +157,6 @@ fn call_target(table: &mlua::Table) -> std::result::Result /// How one yielded value parsed at the resume boundary. #[derive(Debug)] -#[expect( - clippy::large_enum_variant, - reason = "a parse lives only until the driver matches it, and most yields are requests" -)] pub enum YieldParse { /// A well-formed request, ready to dispatch. Request(Request), @@ -240,12 +236,12 @@ fn parse_infer(table: &mlua::Table) -> std::result::Result, }, /// `call(target, input?)`: run a contained chain over the target's @@ -182,9 +182,9 @@ pub enum Request { /// dispatch, not the parse, refuses an empty list. Its records /// were validated as the list added them. list: MessageList, - /// The receiver of a handle's `loop`, as its frozen binding; `None` - /// when the round names no handle, and the driver resolves the - /// section's current model. + /// The leading handle of a `models.loop` call, as its frozen + /// binding; `None` when the round names no handle, and the driver + /// resolves the section's current model. binding: Option, }, /// `store.*(...)`: one run-scoped store operation as a leaf yield. diff --git a/crates/runvil-internal/lua/src/protocol/tests/parse.rs b/crates/runvil-internal/lua/src/protocol/tests/parse.rs index fd987c3bd..093781a30 100644 --- a/crates/runvil-internal/lua/src/protocol/tests/parse.rs +++ b/crates/runvil-internal/lua/src/protocol/tests/parse.rs @@ -228,7 +228,8 @@ fn an_infer_with_a_missing_or_non_string_prompt_is_the_calls_error() { #[test] fn an_infer_with_a_wrong_handle_type_is_the_calls_error() { - // The handle is the author-supplied receiver of `h:infer`, so a + // The handle is the author-supplied leading argument of + // `models.infer`, so a // wrong shape is the call's error (pcall-able at the call site), not // a malformed-yield block failure. let lua = Lua::new(); diff --git a/crates/runvil-internal/lua/src/protocol/tests/parse_chat.rs b/crates/runvil-internal/lua/src/protocol/tests/parse_chat.rs index f8c48d010..3aecc72ab 100644 --- a/crates/runvil-internal/lua/src/protocol/tests/parse_chat.rs +++ b/crates/runvil-internal/lua/src/protocol/tests/parse_chat.rs @@ -28,7 +28,7 @@ fn chat_request(lua: &Lua, records: &str) -> mlua::Table { } /// A `chat` request over the list [`chat_request`] builds, with `handle` -/// beside it, as a handle's `loop` yields it. +/// beside it, as a `models.loop` call with a leading handle yields it. fn chat_request_with_handle(lua: &Lua, records: &str, handle: impl IntoLua) -> mlua::Table { lua_table_with( lua, @@ -190,7 +190,8 @@ fn messages_that_are_not_a_list_are_a_malformed_yield() { #[test] fn chat_with_the_loops_handle_holds_its_frozen_binding() { - // A handle's `loop` yields its receiver beside the messages; the + // A `models.loop` call with a leading handle yields that handle beside + // the messages; the // binding is cloned out of the userdata at the parse, so the round runs // on the handle's model rather than the section default. let lua = Lua::new(); diff --git a/crates/runvil-internal/lua/src/scope.rs b/crates/runvil-internal/lua/src/scope.rs index 4ef1295b8..d335e3387 100644 --- a/crates/runvil-internal/lua/src/scope.rs +++ b/crates/runvil-internal/lua/src/scope.rs @@ -158,7 +158,7 @@ impl Offers { } /// The two sets a section entry froze at its first loop round -/// (`models.loop` or a handle's `loop`): the one record of what its model +/// (`models.loop`): the one record of what its model /// was shown. Each holds its tools' bindings under their wire names, in /// merge order, and no wire name appears twice across them. #[derive(Debug, Clone, Default, PartialEq, Eq)] diff --git a/crates/runvil-internal/lua/src/test_support.rs b/crates/runvil-internal/lua/src/test_support.rs index e47037c4d..a3c4b0674 100644 --- a/crates/runvil-internal/lua/src/test_support.rs +++ b/crates/runvil-internal/lua/src/test_support.rs @@ -7,7 +7,6 @@ use std::collections::BTreeMap; use std::sync::{Arc, Mutex}; use mlua::{Function, Lua, MultiValue, Table}; -use runvil_model_client::model::ModelSet; use runvil_types::emitter::Emitter; use runvil_types::event::lifecycle; use runvil_types::packs::PackId; @@ -19,7 +18,6 @@ use crate::bindings::{RosterEntry, ToolBinding, ToolSet}; use crate::coro::MODEL_TOOL_CALL_REGISTRY; use crate::error::{Error, Result}; use crate::error_value::ErrorValue; -use crate::models::ModelSelection; use crate::program::LuaProgram; use crate::protocol::{Answer, render_envelope}; use crate::scope::Offers; @@ -154,7 +152,8 @@ impl LuaProgram { impl SectionVm { /// Installs the section's Engine values, ahead of the Pack preludes, - /// with an empty `var` and a frozen nil `argv`. + /// with an empty `var`, a frozen nil `argv`, and a selection that starts + /// on the `default` binding, if the model set has one. /// /// # Errors /// Returns the [`InjectFailure`] holding the bridging failure if Engine @@ -165,7 +164,9 @@ impl SectionVm { sys: &serde_json::Value, access: &Arc, ) -> std::result::Result { - self.inject_values_with_var(args, sys, access, None, Argv::Frozen(None)) + let set = &self.bound_models; + let selection = Arc::new(Mutex::new(set.binding("default").cloned())); + self.inject_values_with_var(args, sys, access, None, Argv::Frozen(None), &selection) } } @@ -220,19 +221,4 @@ impl SectionVm { pub fn tool_bag_handles(&self) -> (Arc, Arc>) { (Arc::clone(&self.tool_set), Arc::clone(self.offers())) } - - /// Returns a snapshot of the shared model set and the live section - /// model selection. Production reads the run's shared set through the - /// model view instead. - /// - /// # Errors - /// Returns [`Error::Internal`] if the shared model set's mutex is poisoned. - pub fn model_bag_handles(&self) -> Result<(ModelSet, Arc>)> { - let models = self - .bound_models - .lock() - .map_err(|_| Error::internal("model set mutex was poisoned"))? - .clone(); - Ok((models, Arc::clone(self.model_selection()))) - } } diff --git a/crates/runvil-internal/lua/src/test_vm.rs b/crates/runvil-internal/lua/src/test_vm.rs index 1a674e9bb..76c3b1ffa 100644 --- a/crates/runvil-internal/lua/src/test_vm.rs +++ b/crates/runvil-internal/lua/src/test_vm.rs @@ -10,7 +10,7 @@ use std::ops::Deref; use std::sync::{Arc, Mutex}; use mlua::{FromLuaMulti, MultiValue}; -use runvil_model_client::model::ModelSet; +use runvil_model_client::model::{ModelBinding, ModelSet}; use runvil_types::emitter::Emitter; use runvil_types::seed::RunSeed; use runvil_types::tools::{ToolCatalog, ToolId, ToolInfo}; @@ -239,6 +239,8 @@ pub(crate) fn dynamic(vm: &SectionVm) -> Vec { .collect() } +type Selection = Arc>>; + /// A section VM's setup as data. Unset, the VM shares an empty tool and /// model set, reports nowhere under [`TEST_SECTION`], gets a fresh store, /// empty `args` and `sys`, a frozen nil `argv`, and an empty `var`. @@ -378,7 +380,7 @@ impl<'a> TestVm<'a> { /// Builds the VM, failing the test at the first setup step that fails. #[track_caller] pub(crate) fn build(mut self) -> BuiltVm { - let (mut vm, models, access) = self.run_injection(); + let (mut vm, selection, access) = self.run_injection(); vm.install_engine_globals(&self.emitter, self.section) .expect("the Engine globals install"); if self.shims { @@ -391,7 +393,7 @@ impl<'a> TestVm<'a> { } BuiltVm { vm, - models, + selection, access, emitter: self.emitter, section: self.section.to_owned(), @@ -399,15 +401,16 @@ impl<'a> TestVm<'a> { } /// Construction, the limits, the raw-model-id opt-in, and Engine - /// injection, handing back the injected VM beside the model set and - /// access it shares. + /// injection, handing back the VM beside its selection (which starts on + /// the default binding, if any) and the access it shares. #[track_caller] - fn run_injection(&mut self) -> (SectionVm, Arc>, Arc) { + fn run_injection(&mut self) -> (SectionVm, Selection, Arc) { let tools = Arc::new(std::mem::take(&mut self.tools)); - let models = Arc::new(Mutex::new(std::mem::take(&mut self.models))); + let set = Arc::new(std::mem::take(&mut self.models)); + let selection = Arc::new(Mutex::new(set.binding("default").cloned())); let access = self.access.take().unwrap_or_else(fresh_access); let mut fresh = - SectionVm::new_for_section(&test_nonce(), &tools, &models, &self.emitter, self.section) + SectionVm::new_for_section(&test_nonce(), &tools, &set, &self.emitter, self.section) .expect("the test VM builds"); if let Some((memory_bytes, log_events)) = self.limits { fresh @@ -425,18 +428,19 @@ impl<'a> TestVm<'a> { &access, self.var, self.argv, + &selection, ) .expect("the Engine values inject"); - (vm, models, access) + (vm, selection, access) } } -/// A built test VM: the section VM, which it derefs to, beside the model -/// set and access it shares. +/// A built test VM: the section VM, which it derefs to, beside the selection +/// and access it shares. pub(crate) struct BuiltVm { vm: SectionVm, - /// The run's model set the VM shares. - pub(crate) models: Arc>, + /// The section's current-model selection, which injection received. + pub(crate) selection: Selection, /// The access the VM's `store` table operates through. pub(crate) access: Arc, emitter: Emitter, diff --git a/crates/runvil-internal/lua/src/tests-recording.rs b/crates/runvil-internal/lua/src/tests-recording.rs index 4d3f19fb7..20d8050c8 100644 --- a/crates/runvil-internal/lua/src/tests-recording.rs +++ b/crates/runvil-internal/lua/src/tests-recording.rs @@ -9,7 +9,7 @@ use std::sync::Mutex; -use runvil_types::emitter::{DebugMode, Emitter, EventSink}; +use runvil_types::emitter::{Emitter, EventSink}; use runvil_types::event::Event; use runvil_types::tools::ToolId; @@ -145,7 +145,7 @@ impl Recorder { /// A recorder whose emitter reports under `execution`. pub(super) fn for_execution(execution: &str) -> Self { let sink = EventSink::default(); - let emitter = Emitter::root(sink.clone(), execution, DebugMode::Off); + let emitter = Emitter::root(sink.clone(), execution, false); Self { sink, emitter, @@ -161,7 +161,7 @@ impl Recorder { /// A second emitter over the same sink reporting under another /// execution id, for a test that interleaves runs. pub(super) fn emitter_for(&self, execution: &str) -> Emitter { - Emitter::root(self.sink.clone(), execution, DebugMode::Off) + Emitter::root(self.sink.clone(), execution, false) } /// Every event reported so far, in order. @@ -261,5 +261,5 @@ impl Recorder { /// An emitter whose events nobody reads: the silent stand-in a test passes /// where it has nothing to assert about the boundaries. pub(crate) fn null_emitter() -> Emitter { - Emitter::root(EventSink::default(), "lua-test", DebugMode::Off) + Emitter::root(EventSink::default(), "lua-test", false) } diff --git a/crates/runvil-internal/lua/src/tests/section_vm.rs b/crates/runvil-internal/lua/src/tests/section_vm.rs index 4c13da508..d6c2a16a0 100644 --- a/crates/runvil-internal/lua/src/tests/section_vm.rs +++ b/crates/runvil-internal/lua/src/tests/section_vm.rs @@ -109,6 +109,7 @@ fn a_refused_injection_hands_back_the_vm_for_its_teardown() { &fresh_access(), Some(&json!(5)), Argv::Frozen(None), + &Arc::new(Mutex::new(None)), ) .expect_err("a non-object var seed is refused"); assert!( @@ -169,7 +170,7 @@ fn section_vm_value_injection_bypasses_chunk_global_metatables() { let vm = SectionVm::new_for_section( &test_nonce(), &Arc::new(bindings), - &Arc::new(Mutex::new(ModelSet::default())), + &Arc::new(ModelSet::default()), &null_emitter(), "Test", ) diff --git a/crates/runvil-internal/lua/src/tools/tests-parameters.rs b/crates/runvil-internal/lua/src/tools/tests-parameters.rs index abab46800..1cdc21dfb 100644 --- a/crates/runvil-internal/lua/src/tools/tests-parameters.rs +++ b/crates/runvil-internal/lua/src/tools/tests-parameters.rs @@ -39,12 +39,19 @@ fn search_tool() -> ToolInfo { ) } -/// A set declaring `web`, whose catalog holds `web/search`. +/// A set declaring `web`, whose catalog holds `web/search` and +/// `web/replace`, the second marked model-only. pub(super) fn search_set() -> ToolSet { + let model_only = ToolInfo::new( + ToolId::parse("web/replace").expect("a static valid tool id"), + "Replace text the model names", + search_schema(), + ) + .model_only_call(true); ToolSet::new( vec![PackId::parse("web").expect("`web` is a valid Pack id")], BTreeMap::new(), - ToolCatalog::new([search_tool()]).expect("one tool"), + ToolCatalog::new([search_tool(), model_only]).expect("two tools"), ) } @@ -111,16 +118,31 @@ fn assigning_parameters_raises() { } #[test] -fn a_tool_object_holds_exactly_id_description_parameters_and_pack() { +fn a_model_only_tool_reads_true_and_stays_named_in_the_catalog() { + let vm = lua_over(search_set()); + let read: (bool, String) = vm.eval( + "local tool = tools.get('web/replace')\n\ + return tool.model_only_call, tool.id", + ); + assert_eq!( + read, + (true, "web/replace".to_owned()), + "a script sees the flag before it tries to call the tool" + ); +} + +#[test] +fn a_tool_object_holds_exactly_id_description_parameters_pack_and_model_only_call() { let vm = lua_over(search_set()); - let fields: (String, bool, bool, String) = vm.eval( + let fields: (String, bool, bool, String, bool) = vm.eval( "local tool = tools.get('web/search')\n\ return tool.id, type(tool.description) == 'string', \ - type(tool.parameters) == 'table', tool.pack.name", + type(tool.parameters) == 'table', tool.pack.name, tool.model_only_call", ); assert_eq!( fields, - ("web/search".to_owned(), true, true, "web".to_owned()) + ("web/search".to_owned(), true, true, "web".to_owned(), false), + "a catalog tool a Pack did not mark model-only reads false" ); let extra: Option = vm.eval( "local tool = tools.get('web/search')\n\ @@ -131,7 +153,7 @@ fn a_tool_object_holds_exactly_id_description_parameters_and_pack() { end\n\ return nil", ); - assert_eq!(extra, None, "a tool object reads no field beyond its four"); + assert_eq!(extra, None, "a tool object reads no field beyond its five"); } /// Every string in `value`, keys included, depth first. diff --git a/crates/runvil-internal/lua/src/tools/userdata.rs b/crates/runvil-internal/lua/src/tools/userdata.rs index c4e27ab9f..82487253f 100644 --- a/crates/runvil-internal/lua/src/tools/userdata.rs +++ b/crates/runvil-internal/lua/src/tools/userdata.rs @@ -1,8 +1,9 @@ //! The tool object: one tool as Lua sees it. //! //! Presentation only: the userdata exposes a tool's `id`, its -//! `description`, its `parameters` schema, and its `pack`, the Pack -//! object `tools.packs.get` returns, and stands for its id wherever +//! `description`, its `parameters` schema, its `pack`, the Pack object +//! `tools.packs.get` returns, and whether only a model may call it, and +//! stands for its id wherever //! `tools.offer` and `tools.call` take a tool. The object is frozen and //! methodless, so `description` is always the text the Pack or the //! `tools.adhoc` attrs declared, and each `parameters` read is a fresh @@ -17,35 +18,40 @@ use serde_json::Value as Json; use crate::bindings::ToolBinding; use crate::packs::pack_object; -/// One tool as Lua sees it: `id`, `description`, `parameters`, and -/// `pack`, all read-only. +/// One tool as Lua sees it: `id`, `description`, `parameters`, `pack`, and +/// `model_only_call`, all read-only. #[derive(Debug, Clone, PartialEq, Eq)] pub(crate) struct LuaToolHandle { id: ToolId, description: String, parameters: Json, + model_only_call: bool, } impl LuaToolHandle { /// Builds the object for an adhoc tool from its binding: the id, the - /// declared description, and the parameter schema. + /// declared description, the parameter schema, and whether only a + /// model may call it. #[must_use] pub(crate) fn from_binding(binding: &ToolBinding) -> Self { Self { id: binding.id().clone(), description: binding.description().to_owned(), parameters: binding.schema().clone(), + model_only_call: binding.model_only_call(), } } /// Builds the object for a catalog tool from its info: the id, the - /// declared description, and the parameter schema. + /// declared description, the parameter schema, and whether only a + /// model may call it. #[must_use] pub(crate) fn from_info(info: &ToolInfo) -> Self { Self { id: info.id.clone(), description: info.description.clone(), parameters: info.parameters_schema.clone(), + model_only_call: info.model_only_call, } } @@ -64,6 +70,10 @@ impl UserData for LuaToolHandle { // marker, so `json.encode` gives back `[]`, and nulls read // `json.null`. fields.add_field_method_get("parameters", |lua, this| lua.to_value(&this.parameters)); + // A script reads this before offering a tool whose call it cannot + // make: the model's arguments, not the script's, fit the schema a + // render gave it. + fields.add_field_method_get("model_only_call", |_, this| Ok(this.model_only_call)); fields.add_field_method_get("pack", |lua, this| { pack_object(lua, &this.id.pack().to_string()) }); diff --git a/crates/runvil-internal/lua/src/vm.rs b/crates/runvil-internal/lua/src/vm.rs index 4419d9cd5..9a2064297 100644 --- a/crates/runvil-internal/lua/src/vm.rs +++ b/crates/runvil-internal/lua/src/vm.rs @@ -24,7 +24,7 @@ use std::sync::atomic::{AtomicBool, AtomicU32, AtomicUsize, Ordering}; use std::sync::{Arc, Mutex}; use mlua::{Lua, LuaOptions, StdLib}; -use runvil_model_client::model::ModelSet; +use runvil_model_client::model::{ModelBinding, ModelSet}; use runvil_types::emitter::Emitter; use runvil_types::event::lifecycle; use runvil_types::untrusted::GuardNonce; @@ -37,7 +37,7 @@ use crate::error::{Error, Result}; use crate::hardening::{CancelHook, harden, install_cancel_hook}; use crate::iteration::install_deterministic_iteration; use crate::messages::MessageList; -use crate::models::{ModelSelection, RawModelIds}; +use crate::models::RawModelIds; use crate::program::LuaProgram; use crate::scope::Offers; use crate::walk_budget::HeapCeiling; @@ -46,9 +46,7 @@ pub use adhoc::AdhocTools; pub use inject::InjectFailure; pub use run::CoroStep; pub(crate) use run::scalar_return; -pub use state::{ - current_catalog_packs, current_dynamic_set, current_static_set, resolve_model_binding, -}; +pub use state::{current_catalog_packs, current_dynamic_set, current_static_set}; /// Default per-VM Lua heap ceiling, matching the executor's `RunLimits`. pub(crate) const DEFAULT_LUA_MEMORY_BYTES: usize = 64 * 1024 * 1024; @@ -85,12 +83,16 @@ pub struct Uninjected; /// A section VM's stage once Engine injection ran, holding the chain /// step's VFS access capability, which the `store` table's closures share, /// so every store op is attributed to the identity the executor installed, -/// and the live `sys` mirror injection builds from the sealed `sys`. +/// the live `sys` mirror injection builds from the sealed `sys`, and the +/// selection that holds the chain's current model. #[derive(Debug)] pub struct Injected { access: Arc, /// Live sealed `sys` JSON, mirrored for prose snapshots. sys_live: Arc>, + /// The current model: what `models.use` stored, starting on the chain's + /// starting value. + selection: Arc>>, } /// One hardened, isolated Lua VM for a section's complete lifecycle. @@ -120,13 +122,11 @@ pub struct SectionVm { /// run catalog. Shared with the run, not snapshotted; nothing writes /// it during a run. pub(crate) tool_set: Arc, - /// The run's shared model set: the frontmatter's filled roles plus the - /// prompt-wide default. Shared for the same reason (`models.default`). - pub(crate) bound_models: Arc>, + /// The run's shared model set: the five filled roles, built once with + /// the run and never written during it. + pub(crate) bound_models: Arc, /// The section's offers, read by the executor's scope path. offers: Arc>, - /// The section's model selection, read by the executor's scope path. - model_selection: Arc>, /// Set by Lua `jump` before it aborts the current chunk. jump_slot: Arc>>, /// How many adhoc tool handlers are running on this VM; `jump` refuses @@ -157,8 +157,8 @@ pub struct SectionVm { /// (hooks are per-coroutine in PUC Lua). cancel_hook: CancelHook, /// The raw-model-id fallback, resolved whenever the caller passes an - /// application-state snapshot (`RunContext::with_ui`): then `models.get` - /// resolves an undeclared alias as a raw gateway catalog model id. Set + /// application-state snapshot (`RunConfig::with_ui`): then `models.get` + /// resolves a label outside the five roles as a raw gateway model id. Set /// by [`allow_raw_model_ids`](SectionVm::allow_raw_model_ids) before /// Engine injection; refused everywhere else. raw_model_ids: RawModelIds, @@ -206,9 +206,8 @@ impl SectionVm { let mut vm = Self { lua, tool_set: Arc::new(ToolSet::default()), - bound_models: Arc::new(Mutex::new(ModelSet::default())), + bound_models: Arc::new(ModelSet::default()), offers: Arc::new(Mutex::new(Offers::default())), - model_selection: Arc::new(Mutex::new(ModelSelection::new())), jump_slot: Arc::new(Mutex::new(None)), adhoc_handler_depth: Arc::new(AtomicU32::new(0)), loop_running: Arc::new(AtomicBool::new(false)), @@ -243,17 +242,16 @@ impl SectionVm { /// /// The sets are the run's own handles, not snapshots: the run catalog /// and the filled roles back the validating `tools`/`models` tables that - /// [`inject_values_with_var`](Self::inject_values_with_var) installs, - /// and the prompt-wide fact a section records (`models.default`) lands - /// where every later section sees it. Nothing writes the tool set - /// during a run. H1 is section 0 on this same path. + /// [`inject_values_with_var`](Self::inject_values_with_var) installs. + /// Nothing writes either set during a run. H1 is section 0 on this same + /// path. /// /// # Errors /// Returns [`Error::LuaRuntime`] if the VM cannot be built or hardened. pub fn new_for_section( nonce: &GuardNonce, tools: &Arc, - models: &Arc>, + models: &Arc, emitter: &Emitter, section: &str, ) -> Result { @@ -264,11 +262,11 @@ impl SectionVm { } /// Opts the VM into the raw-model-id fallback, on whenever the caller - /// passes an application-state snapshot (`RunContext::with_ui`): `models.get` - /// resolves an undeclared alias as a raw gateway catalog model id. - /// Engine injection's `models` table install reads the flag, so only an - /// uninjected VM takes it. A run without a snapshot never calls it, so - /// it keeps strict declared-alias resolution. + /// passes an application-state snapshot (`RunConfig::with_ui`): `models.get` + /// resolves a label outside the five roles as a raw gateway catalog model + /// id. Engine injection's `models` table install reads the flag, so only + /// an uninjected VM takes it. A run without a snapshot never calls it, so + /// `models.get` accepts only the five roles. pub fn allow_raw_model_ids(&mut self) { self.raw_model_ids = RawModelIds::Resolved; } @@ -331,12 +329,6 @@ impl SectionVm { &self.adhoc_tools } - /// The section's model selection: what `models.use` recorded. - #[must_use] - pub fn model_selection(&self) -> &Arc> { - &self.model_selection - } - /// The section's one conversation, which the `messages` global holds /// from Engine injection until the VM drops. #[must_use] diff --git a/crates/runvil-internal/lua/src/vm/inject.rs b/crates/runvil-internal/lua/src/vm/inject.rs index 69a07dc4b..ab9b81c9f 100644 --- a/crates/runvil-internal/lua/src/vm/inject.rs +++ b/crates/runvil-internal/lua/src/vm/inject.rs @@ -4,6 +4,7 @@ use std::sync::{Arc, Mutex}; +use runvil_model_client::model::ModelBinding; use runvil_types::emitter::Emitter; use runvil_vfs::Access; use serde_json::Value as Json; @@ -66,6 +67,9 @@ impl SectionVm { /// executor reads back with [`argv_json`](SectionVm::argv_json)), frozen /// for every other section. /// + /// `selection` holds the current model: `models.use` writes it, and + /// [`current_model`](SectionVm::current_model) reads it. + /// /// # Errors /// Returns the [`InjectFailure`] holding the bridging failure if Engine /// values cannot be bridged. @@ -80,8 +84,9 @@ impl SectionVm { access: &Arc, initial_var: Option<&Json>, argv: Argv<'_>, + selection: &Arc>>, ) -> std::result::Result { - if let Err(error) = self.install_values(args, sys, initial_var, argv) { + if let Err(error) = self.install_values(args, sys, initial_var, argv, selection) { return Err(InjectFailure { vm: Box::new(self), error, @@ -90,6 +95,7 @@ impl SectionVm { Ok(self.injected(Injected { access: Arc::clone(access), sys_live: Arc::new(Mutex::new(sys.clone())), + selection: Arc::clone(selection), })) } @@ -103,6 +109,7 @@ impl SectionVm { sys: &Json, initial_var: Option<&Json>, argv: Argv<'_>, + selection: &Arc>>, ) -> Result<()> { let globals = self.lua.globals(); globals.raw_set("args", args).map_err(Error::lua)?; @@ -126,7 +133,7 @@ impl SectionVm { &self.lua, &globals, &self.bound_models, - &self.model_selection, + selection, self.raw_model_ids, )?; install_messages(&self.lua, &self.messages)?; @@ -140,7 +147,6 @@ impl SectionVm { tool_set, bound_models, offers, - model_selection, jump_slot, adhoc_handler_depth, loop_running, @@ -160,7 +166,6 @@ impl SectionVm { tool_set, bound_models, offers, - model_selection, jump_slot, adhoc_handler_depth, loop_running, diff --git a/crates/runvil-internal/lua/src/vm/install.rs b/crates/runvil-internal/lua/src/vm/install.rs index 8c4d3b4a4..7579576ad 100644 --- a/crates/runvil-internal/lua/src/vm/install.rs +++ b/crates/runvil-internal/lua/src/vm/install.rs @@ -134,7 +134,9 @@ impl SectionVm { /// namespace) through `install_shim_prelude`, which also puts /// `models.loop` and `models.compact` on the `models` table. /// `max_tool_iterations` is the run's resolved round cap, which every - /// `models.loop` and model handle `loop` call on this VM spends. + /// `models.loop` and model handle `loop` call on this VM spends. The + /// `tasks.fanout` shim reads and writes the chain's current model + /// through captures over the selection injection received. /// /// # Errors /// Returns [`Error::LuaRuntime`] if the shim prelude cannot install. @@ -142,6 +144,7 @@ impl SectionVm { install_shim_prelude( &self.lua, max_tool_iterations, + &self.state.selection, &self.adhoc_handler_depth, &self.loop_running, &self.cancel_hook, diff --git a/crates/runvil-internal/lua/src/vm/run.rs b/crates/runvil-internal/lua/src/vm/run.rs index 4dacbbd53..0f73dfdca 100644 --- a/crates/runvil-internal/lua/src/vm/run.rs +++ b/crates/runvil-internal/lua/src/vm/run.rs @@ -213,6 +213,7 @@ impl SectionVm { | ErrorKind::ModelRequired | ErrorKind::OutOfScopeTool | ErrorKind::UnboundTool + | ErrorKind::ModelOnlyTool | ErrorKind::Tool | ErrorKind::TaskNotOwned | ErrorKind::TaskConsumed diff --git a/crates/runvil-internal/lua/src/vm/state.rs b/crates/runvil-internal/lua/src/vm/state.rs index 5b1c75447..0c74bc323 100644 --- a/crates/runvil-internal/lua/src/vm/state.rs +++ b/crates/runvil-internal/lua/src/vm/state.rs @@ -1,11 +1,11 @@ //! Section VM state reads: `var`, bare globals, `argv`, the tool call //! counts, the frozen sets' entries, the section's offered tools and the -//! catalog Packs they come from, and its effective model binding. +//! catalog Packs they come from, and its current model. use std::sync::{Arc, Mutex}; use mlua::{LuaSerdeExt, Value}; -use runvil_model_client::model::{ModelBinding, ModelView}; +use runvil_model_client::model::ModelBinding; use runvil_types::packs::PackId; use runvil_types::tools::{ToolId, ToolInfo}; use serde_json::Value as Json; @@ -13,7 +13,6 @@ use serde_json::Value as Json; use super::{AdhocTools, SectionVm}; use crate::bindings::{ToolBinding, ToolSet}; use crate::error::{Error, Result}; -use crate::models::ModelSelection; use crate::packs::is_adhoc; use crate::scope::{OfferMode, OfferTarget, OfferedTool, Offers, RenderedTools, ToolCallCounts}; use crate::tools::install_tool_call_counts as install_tool_call_counts_impl; @@ -110,6 +109,20 @@ impl SectionVm { pub fn rendered_tools(&self) -> Option> { self.messages.rendered_tools() } + + /// Returns the section's current model: the binding `models.use` last + /// stored, else the model the section started on. `None` means the caller + /// supplied no model. + /// + /// # Errors + /// Returns [`Error::Internal`] if the selection's mutex is poisoned. + pub fn current_model(&self) -> Result> { + self.state + .selection + .lock() + .map(|selection| selection.clone()) + .map_err(|_| Error::internal("model selection lock was poisoned")) + } } /// Reads the tools the section's static offers name without mutating the @@ -240,36 +253,3 @@ fn expanded( } Ok(tools) } - -/// Reads the section's effective model binding through the run's model view -/// without mutating the model selection: the H2 `models.use` selection with -/// its options applied, else the prompt-wide `models.default` baseline. -/// -/// # Errors -/// Returns [`Error::Internal`] if the model selection's mutex or the shared -/// model set is poisoned, or the selected alias has no frozen binding. -pub fn resolve_model_binding( - bindings: &dyn ModelView, - runtime: &Mutex, -) -> Result> { - let selection = { - let runtime = runtime - .lock() - .map_err(|_| Error::internal("model selection lock was poisoned"))?; - runtime - .selection() - .map(|(alias, options)| (alias.as_str().to_owned(), options)) - }; - let frozen = |alias: &str| -> Result { - bindings - .binding(alias)? - .ok_or_else(|| Error::internal(format!("model alias {alias:?} has no frozen binding"))) - }; - match selection { - Some((alias, options)) => Ok(Some(options.apply(frozen(&alias)?))), - None => match bindings.default_alias()? { - Some(alias) => Ok(Some(frozen(&alias)?)), - None => Ok(None), - }, - } -} diff --git a/crates/runvil-internal/model-client/src/client/tests.rs b/crates/runvil-internal/model-client/src/client/tests.rs index 4136dcd79..e6153f3af 100644 --- a/crates/runvil-internal/model-client/src/client/tests.rs +++ b/crates/runvil-internal/model-client/src/client/tests.rs @@ -150,10 +150,6 @@ fn from_parts_refuses_a_blank_id_a_blank_name_and_non_object_arguments() { CompletionErrorKind::InvalidParts, "{label} must be refused" ); - assert!( - !error.is_retryable(), - "{label}: rebuilding the same parts fails the same way" - ); } let call = ToolCall::from_parts("call_1", "fetch", object()).expect("a whole call is accepted"); assert_eq!((call.id(), call.name()), ("call_1", "fetch")); @@ -169,7 +165,6 @@ fn from_result_refuses_an_empty_batch_and_duplicate_ids_and_accepts_text() { let twice = CompletionResult::ToolCalls(vec![call.clone(), call.clone()]); let duplicate = Completion::from_result(twice, "m").expect_err("a duplicate id is refused"); assert_eq!(duplicate.kind(), CompletionErrorKind::InvalidParts); - assert!(!duplicate.is_retryable()); assert!( duplicate.to_string().contains("\"call_1\""), "the message names the repeated id: {duplicate}" diff --git a/crates/runvil-internal/model-client/src/lib.rs b/crates/runvil-internal/model-client/src/lib.rs index d8d10e93b..30f97cc10 100644 --- a/crates/runvil-internal/model-client/src/lib.rs +++ b/crates/runvil-internal/model-client/src/lib.rs @@ -10,7 +10,7 @@ //! [`ModelCatalog`](runvil_types::models::ModelCatalog) the //! caller supplies, the validated [`ModelId`](runvil_types::models::ModelId) //! identity, and the -//! [`model::ModelBinding`]/[`model::ModelSet`]/[`model::ModelView`] types +//! [`model::ModelBinding`]/[`model::ModelSet`] types //! model selections resolve and freeze through, with //! [`model::CompletionError`] as the failure a round reports. //! diff --git a/crates/runvil-internal/model-client/src/model.rs b/crates/runvil-internal/model-client/src/model.rs index 76c4d3a66..1fbb6ef5e 100644 --- a/crates/runvil-internal/model-client/src/model.rs +++ b/crates/runvil-internal/model-client/src/model.rs @@ -1,23 +1,20 @@ -//! Prompt-local model bindings: catalog, bind/use declarations, and invocation. +//! Prompt-local model bindings: catalog, role bindings, and invocation. //! //! The caller supplies the //! [`ModelCatalog`](runvil_types::models::ModelCatalog); building it happens -//! outside this crate. The prompt's frontmatter declares each model role, prepare -//! resolves every role against that catalog under the role's hard keywords -//! and context minimum, and the run builds its binding set from those -//! bindings before any Lua runs. H2 `models.use` selects at most one binding -//! per section; `models.default` supplies the prompt-wide default, -//! conventionally from H1, for sections that omit `models.use`. Model-facing -//! sections with neither binding fail with a -//! model-binding failure surfaced through the run error the caller receives. - +//! outside this crate. Building a run fills each of the five model roles +//! from the caller's role map into its [`ModelSet`], once, before any Lua +//! runs, and the run's first step checks the prompt's declared requirements +//! against the model that fills each role. `models.use` selects one binding for +//! a section, and a section that never calls it runs on the `default` role. +//! A model-facing section with no model fails with a model-binding failure +//! surfaced through the run error the caller receives. mod error; mod options; pub use error::{CompletionError, CompletionErrorKind}; pub use options::{ - CompletionOptions, ModelBinding, ModelInvocation, ModelSet, ModelSetError, ModelView, - Temperature, TemperatureError, + CompletionOptions, ModelBinding, ModelInvocation, ModelSet, Temperature, TemperatureError, }; // `runvil-lua` names the model id through this path; the rest of the catalog // vocabulary is imported from `runvil-types` directly. diff --git a/crates/runvil-internal/model-client/src/model/error-tests.rs b/crates/runvil-internal/model-client/src/model/error-tests.rs index cd74d2ee9..bc7d4f9fa 100644 --- a/crates/runvil-internal/model-client/src/model/error-tests.rs +++ b/crates/runvil-internal/model-client/src/model/error-tests.rs @@ -1,5 +1,5 @@ -//! Tests for the failure vocabulary: retryability per kind, the fixed -//! phrases, the specifics that extend them, and the detail channel. +//! Tests for the failure vocabulary: the tags, the fixed phrases, the +//! specifics that extend them, and the detail channel. use super::*; @@ -19,26 +19,6 @@ const ALL_KINDS: [CompletionErrorKind; 13] = [ CompletionErrorKind::InvalidParts, ]; -#[test] -fn retryability_is_fixed_per_kind() { - let retryable = [ - CompletionErrorKind::RateLimited, - CompletionErrorKind::Overloaded, - CompletionErrorKind::Timeout, - CompletionErrorKind::Transport, - CompletionErrorKind::ServerError, - CompletionErrorKind::MalformedResponse, - ]; - for kind in ALL_KINDS { - let error = CompletionError::new(kind, "message"); - assert_eq!( - error.is_retryable(), - retryable.contains(&kind), - "retryability of {kind:?}" - ); - } -} - #[test] fn each_kind_tags_as_its_snake_case_name() { let tags: Vec<&str> = ALL_KINDS.iter().map(|kind| kind.tag()).collect(); @@ -152,7 +132,6 @@ fn context_overflow_holds_the_counts() { let error = CompletionError::context_overflow(Some(10), None, "too big"); assert_eq!(error.kind(), CompletionErrorKind::ContextOverflow); assert_eq!(error.overflow(), (Some(10), None)); - assert!(!error.is_retryable()); } #[test] @@ -181,7 +160,6 @@ fn a_timeout_and_a_transport_failure_are_built_from_a_kind_and_a_source() { "deadline", )); assert_eq!(timed_out.kind(), CompletionErrorKind::Timeout); - assert!(timed_out.is_retryable()); assert_eq!( timed_out.to_string(), "the model backend did not answer in time" @@ -199,7 +177,6 @@ fn a_timeout_and_a_transport_failure_are_built_from_a_kind_and_a_source() { ) .with_source(std::io::Error::other("connection refused")); assert_eq!(refused.kind(), CompletionErrorKind::Transport); - assert!(refused.is_retryable()); } #[test] @@ -225,7 +202,6 @@ fn a_specific_extends_the_phrase_after_a_colon_and_never_adds_detail() { ); assert_eq!(malformed.message(), malformed.to_string()); assert_eq!(malformed.detail(), None); - assert!(malformed.is_retryable()); let empty = CompletionError::specific( CompletionErrorKind::EmptyReply, @@ -238,7 +214,6 @@ fn a_specific_extends_the_phrase_after_a_colon_and_never_adds_detail() { reasoning content was present but ignored" ); assert_eq!(empty.finish_reason(), Some("stop")); - assert!(!empty.is_retryable()); } #[test] @@ -250,7 +225,6 @@ fn the_shared_empty_reply_owner_builds_the_phrase_and_the_reasoning_specific() { assert_eq!(plain.kind(), CompletionErrorKind::EmptyReply); assert_eq!(plain.to_string(), CompletionErrorKind::EmptyReply.phrase()); assert_eq!(plain.finish_reason(), None); - assert!(!plain.is_retryable()); let reasoned = CompletionError::empty_reply(true).with_finish_reason("stop"); assert_eq!( diff --git a/crates/runvil-internal/model-client/src/model/error.rs b/crates/runvil-internal/model-client/src/model/error.rs index b3a932197..ba6d80a77 100644 --- a/crates/runvil-internal/model-client/src/model/error.rs +++ b/crates/runvil-internal/model-client/src/model/error.rs @@ -19,9 +19,9 @@ const REASONING_IGNORED: &str = "reasoning content was present but ignored"; /// /// The kinds form a closed set, so a caller can branch on the kind alone. /// The caller that performs a round maps every failure into one of these -/// kinds. Each kind is either retryable or permanent (see -/// [`CompletionError::is_retryable`]). The enum is `#[non_exhaustive]`, so -/// a `match` on it needs a wildcard arm. +/// kinds. Each kind is either retryable or permanent, as its own doc says. +/// The enum is `#[non_exhaustive]`, so a `match` on it needs a wildcard +/// arm. #[non_exhaustive] #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum CompletionErrorKind { @@ -107,18 +107,6 @@ impl CompletionErrorKind { CompletionErrorKind::InvalidParts => "invalid_parts", } } - - fn is_retryable(self) -> bool { - matches!( - self, - CompletionErrorKind::RateLimited - | CompletionErrorKind::Overloaded - | CompletionErrorKind::Timeout - | CompletionErrorKind::Transport - | CompletionErrorKind::ServerError - | CompletionErrorKind::MalformedResponse - ) - } } /// The error a model round or a catalog fetch fails with. @@ -259,15 +247,6 @@ impl CompletionError { self.kind } - /// Returns `true` when retrying may succeed. The answer depends only on - /// the kind. `RateLimited`, `Overloaded`, `Timeout`, `Transport`, - /// `ServerError`, and `MalformedResponse` are retryable, and the rest - /// are permanent. - #[must_use] - pub fn is_retryable(&self) -> bool { - self.kind.is_retryable() - } - /// Returns the token counts that a context overflow reported, as /// `(prompt_tokens, window)`. A count is `None` when the provider /// omitted it, and both are `None` for every kind other than diff --git a/crates/runvil-internal/model-client/src/model/options-tests.rs b/crates/runvil-internal/model-client/src/model/options-tests.rs index 701f94720..63f1199e8 100644 --- a/crates/runvil-internal/model-client/src/model/options-tests.rs +++ b/crates/runvil-internal/model-client/src/model/options-tests.rs @@ -1,44 +1,8 @@ -//! Tests for the model set's read view and the temperature range: the -//! poisoned and healthy view methods, and the finite range check. +//! Tests for the model set's total equivalence, the binding accessors, and the +//! finite temperature range check. use super::*; -/// A model set whose mutex a panicking thread poisoned. -fn poisoned_set() -> Mutex { - let set = Mutex::new(ModelSet::default()); - std::thread::scope(|scope| { - let _ = scope - .spawn(|| { - let _guard = set.lock(); - panic!("poison the set"); - }) - .join(); - }); - set -} - -#[test] -fn a_poisoned_model_set_reports_the_lock_error_from_every_view_method() { - let set = poisoned_set(); - assert!(set.lock().is_err(), "a poisoned set cannot be read"); - assert_eq!(ModelView::default_alias(&set), Err(ModelSetError)); - assert_eq!(set.binding("writer"), Err(ModelSetError)); - assert_eq!(ModelSetError.to_string(), "model set mutex was poisoned"); -} - -#[test] -fn a_healthy_model_set_answers_every_view_method() { - let set = Mutex::new(ModelSet::default()); - assert!( - set.lock() - .expect("a healthy set locks") - .bindings() - .is_empty() - ); - assert_eq!(ModelView::default_alias(&set), Ok(None)); - assert_eq!(set.binding("writer"), Ok(None)); -} - #[test] fn a_model_set_compares_as_a_total_equivalence() { fn assert_eq_impl() {} diff --git a/crates/runvil-internal/model-client/src/model/options.rs b/crates/runvil-internal/model-client/src/model/options.rs index 23a38a727..66deddd73 100644 --- a/crates/runvil-internal/model-client/src/model/options.rs +++ b/crates/runvil-internal/model-client/src/model/options.rs @@ -2,7 +2,6 @@ //! prompt-local bindings, and completion options. use std::num::NonZeroU32; -use std::sync::Mutex; use runvil_types::models::{ModelId, ModelInfo, ThinkingMode}; @@ -117,10 +116,6 @@ pub struct ModelBinding { id: ModelId, invocation: ModelInvocation, context: NonZeroU32, - /// The bound role's full keyword set (the closed frontmatter - /// vocabulary, kebab-case), exposed on the Lua handle as - /// `capabilities`. Empty for bindings built outside a role fill. - capabilities: Vec, /// The model's provider id, when the catalog names one. provider: Option, /// The catalog's description of the bound model, when the binding was @@ -147,7 +142,6 @@ impl ModelBinding { id, invocation, context, - capabilities: Vec::new(), provider: None, catalog: None, } @@ -169,14 +163,6 @@ impl ModelBinding { self } - /// Sets the bound role's capability keywords, which Lua code reads as the - /// `capabilities` field of the model handle. - #[must_use] - pub fn with_capabilities(mut self, capabilities: Vec) -> Self { - self.capabilities = capabilities; - self - } - /// Replaces the binding's request settings. #[must_use] pub fn with_invocation(mut self, invocation: ModelInvocation) -> Self { @@ -184,19 +170,13 @@ impl ModelBinding { self } - /// Returns the bound role's capability keywords. - #[must_use] - pub fn capabilities(&self) -> &[String] { - &self.capabilities - } - /// Returns the exact prompt-local alias. #[must_use] pub fn alias(&self) -> &str { &self.alias } - /// Returns the declared capability description. + /// Returns the description of the bound model. #[must_use] pub fn description(&self) -> &str { &self.description @@ -369,100 +349,32 @@ impl CompletionOptions { } } -/// The run's model set: the prompt-level bindings built at run construction -/// from prepare's role bindings plus the prompt-wide `default` alias. +/// The run's model set: the five role bindings the run builds once, +/// shared unchanged by every section VM. #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct ModelSet { - /// The bindings in declaration order. + /// The bindings in the order they were pushed. bindings: Vec, - /// The prompt-wide default alias set by `models.default`, if any; read - /// through [`default_alias`](ModelSet::default_alias), since an - /// inherent `default()` would shadow `Default::default()`. - default: Option, } impl ModelSet { - /// Returns bindings in declaration order. + /// Returns bindings in the order they were pushed. #[must_use] pub fn bindings(&self) -> &[ModelBinding] { &self.bindings } - /// Returns the binding for `alias`, if it was declared. + /// Returns the binding for `alias`, if the set holds one. #[must_use] pub fn binding(&self, alias: &str) -> Option<&ModelBinding> { self.bindings.iter().find(|binding| binding.alias == alias) } - /// Returns the prompt-wide default alias set by `models.default`, if - /// any. - #[must_use] - pub fn default_alias(&self) -> Option<&str> { - self.default.as_deref() - } - - /// Appends `binding` after every binding already declared. + /// Appends `binding` after every binding already in the set. pub fn push(&mut self, binding: ModelBinding) { self.bindings.push(binding); } - - /// Sets the prompt-wide default alias, replacing any set before. - pub fn set_default_alias(&mut self, alias: String) { - self.default = Some(alias); - } } - -/// The run's model-set lock was poisoned. -/// -/// This is the run's own mutex failing, not a model failure, so it has no -/// [`CompletionErrorKind`](crate::model::CompletionErrorKind) and no retry -/// advice. The Engine and Lua crates map it to their Lua error, and the -/// facade does not re-export it. -#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] -#[error("model set mutex was poisoned")] -#[non_exhaustive] -pub struct ModelSetError; - -/// The read-only view over the run's [`ModelSet`]. -/// -/// The run holds one `ModelSet` behind a mutex and hands every section VM a -/// handle to it. This trait exposes reads only, so a reader cannot change the -/// set through it. The one write any Engine path performs is `models.default`, -/// which sets the default once and refuses a second, different label. Every -/// method locks briefly and returns an owned snapshot: a mutex guard cannot -/// outlive the call. -pub trait ModelView: Send + Sync { - /// Returns the prompt-wide default alias set by `models.default`, if any. - /// - /// # Errors - /// Returns [`ModelSetError`] if the set's mutex is poisoned. - fn default_alias(&self) -> Result, ModelSetError>; - - /// Returns an owned clone of the binding for `alias`, if it was - /// declared. - /// - /// # Errors - /// Returns [`ModelSetError`] if the set's mutex is poisoned. - fn binding(&self, alias: &str) -> Result, ModelSetError>; -} - -/// Maps a poisoned set lock to [`ModelSetError`]. -fn lock_model_set( - set: &Mutex, -) -> Result, ModelSetError> { - set.lock().map_err(|_| ModelSetError) -} - -impl ModelView for Mutex { - fn default_alias(&self) -> Result, ModelSetError> { - Ok(lock_model_set(self)?.default_alias().map(str::to_owned)) - } - - fn binding(&self, alias: &str) -> Result, ModelSetError> { - Ok(lock_model_set(self)?.binding(alias).cloned()) - } -} - #[cfg(test)] #[path = "options-tests.rs"] mod tests; diff --git a/crates/runvil-internal/model-client/src/model/tests.rs b/crates/runvil-internal/model-client/src/model/tests.rs index 3f745f4a4..4475d7202 100644 --- a/crates/runvil-internal/model-client/src/model/tests.rs +++ b/crates/runvil-internal/model-client/src/model/tests.rs @@ -80,29 +80,30 @@ fn binding_construction_is_atomic_with_context() { } #[test] -fn a_model_set_grows_by_push_and_takes_its_default_alias_by_setter() { +fn a_model_set_grows_by_push_and_finds_a_binding_by_alias() { let mut set = ModelSet::default(); assert!(set.bindings().is_empty()); - assert_eq!(set.default_alias(), None); - set.push(binding("writer")); - set.push(binding("critic")); - set.set_default_alias("writer".to_owned()); + assert_eq!(set.binding("default"), None); + set.push(binding("default")); + set.push(binding("strong")); assert_eq!( set.bindings() .iter() .map(ModelBinding::alias) .collect::>(), - ["writer", "critic"], - "push keeps declaration order" + ["default", "strong"], + "push keeps the order bindings were added" ); - assert_eq!(set.default_alias(), Some("writer")); + assert_eq!( + set.binding("strong").map(ModelBinding::alias), + Some("strong") + ); + assert_eq!(set.binding("fast"), None); let mut expected = ModelSet::default(); - expected.push(binding("writer")); - expected.push(binding("critic")); - expected.set_default_alias("writer".to_owned()); + expected.push(binding("default")); + expected.push(binding("strong")); assert_eq!(set, expected); } - #[test] fn completion_options_read_back_every_field_they_were_built_with() { let bare = CompletionOptions::new("analyst"); diff --git a/crates/runvil-internal/parser/src/contract.rs b/crates/runvil-internal/parser/src/contract.rs index d12605d37..51f80913f 100644 --- a/crates/runvil-internal/parser/src/contract.rs +++ b/crates/runvil-internal/parser/src/contract.rs @@ -3,11 +3,10 @@ //! The YAML is the whole contract: Packs install, models declare, args //! type. Parsing validates the static shape - each `uses:` entry a bare //! Pack id or vendor/name, no Pack id declared twice, the alias grammar -//! on map keys, the closed -//! model-keyword vocabulary, arg name and type sanity - and exposes the -//! FULL declaration on the parsed [`Prompt`](crate::Prompt); satisfying -//! the declaration against the caller's environment is prepare's job, -//! never the parser's. +//! on arg names, the five model role names, arg name and type sanity - and +//! exposes the FULL declaration on the parsed [`Prompt`](crate::Prompt); +//! checking the declaration against the caller's environment is the job of +//! the run's first step, never the parser's. //! //! `args` and `models` are defined in submodules; this root owns the //! duplicate-Pack check and the map deserializer the two map keys share. @@ -17,6 +16,7 @@ use std::collections::{BTreeMap, BTreeSet}; use std::fmt; use std::marker::PhantomData; +use runvil_types::models::ModelRole; use runvil_types::packs::{PackId, PackRef}; use serde::de::{self, MapAccess, Visitor}; use serde::{Deserialize, Deserializer}; @@ -28,13 +28,14 @@ mod models; mod tests; pub use args::{ArgDecl, ArgType, ArgsDecl, ArgsDeclIter}; -pub use models::{ModelKeyword, ModelRole, ModelRoles, ModelRolesIter}; +pub(crate) use models::RoleMap; +pub use models::RoleRequirements; /// The prompt-local alias grammar: `[A-Za-z][A-Za-z0-9_-]{0,63}`. /// -/// Model labels and args field names are prompt-local and never global -/// names. (The same rule sits in `runvil-lua`'s `alias` module, the -/// run-time counterpart to this parse-time check.) +/// Args field names are prompt-local and never global names. (The same rule +/// sits in `runvil-lua`'s `alias` module, the run-time counterpart to this +/// parse-time check.) fn is_valid_alias(alias: &str) -> bool { let bytes = alias.as_bytes(); (1..=64).contains(&bytes.len()) @@ -47,19 +48,55 @@ fn is_valid_alias(alias: &str) -> bool { /// How one contract map's keys are described in its errors. #[derive(Clone, Copy)] struct ContractKeys { - /// The key kind (`model role label`), for error messages. + /// The key kind (`model role`), for error messages. what: &'static str, } -/// Deserializes a contract map (`models`, `args`): string keys -/// validated against the alias grammar and `keys`, values deserialized as -/// `T`, duplicates rejected. -fn deserialize_contract_map<'de, D, T>( +/// A key of a contract map: how it is checked and how an error names it. +trait ContractKey: Ord { + /// Refuses a key the map's grammar does not allow. + fn check(&self, what: &str) -> Result<(), String>; + + /// The key as an error names it. + fn shown(&self) -> String; +} + +impl ContractKey for String { + fn check(&self, what: &str) -> Result<(), String> { + if is_valid_alias(self) { + Ok(()) + } else { + Err(format!( + "invalid {what} `{self}`: expected [A-Za-z][A-Za-z0-9_-]{{0,63}}" + )) + } + } + + fn shown(&self) -> String { + self.clone() + } +} + +impl ContractKey for ModelRole { + /// A role name is checked while it deserializes, against the five names. + fn check(&self, _what: &str) -> Result<(), String> { + Ok(()) + } + + fn shown(&self) -> String { + self.as_str().to_owned() + } +} + +/// Deserializes a contract map (`models`, `args`): keys of type `K` checked +/// by [`ContractKey`], values deserialized as `T`, duplicates rejected. +fn deserialize_contract_map<'de, D, K, T>( deserializer: D, keys: ContractKeys, -) -> Result, D::Error> +) -> Result, D::Error> where D: Deserializer<'de>, + K: ContractKey + Deserialize<'de>, T: Deserialize<'de>, { deserializer.deserialize_map(MapVisitor { @@ -70,18 +107,19 @@ where /// The visitor behind [`deserialize_contract_map`]: a streaming map walk so /// rejections keep their source position. -struct MapVisitor { +struct MapVisitor { /// The checks the map's keys get. keys: ContractKeys, - /// The value type, without ownership or variance claims. - _marker: PhantomData T>, + /// The key and value types, without ownership or variance claims. + _marker: PhantomData (K, T)>, } -impl<'de, T> Visitor<'de> for MapVisitor +impl<'de, K, T> Visitor<'de> for MapVisitor where + K: ContractKey + Deserialize<'de>, T: Deserialize<'de>, { - type Value = BTreeMap; + type Value = BTreeMap; fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { write!( @@ -96,18 +134,14 @@ where A: MapAccess<'de>, { let ContractKeys { what } = self.keys; - let mut entries: BTreeMap = BTreeMap::new(); - while let Some(key) = map.next_key::()? { - if !is_valid_alias(&key) { - return Err(de::Error::custom(format!( - "invalid {what} `{key}`: expected [A-Za-z][A-Za-z0-9_-]{{0,63}}" - ))); - } + let mut entries: BTreeMap = BTreeMap::new(); + while let Some(key) = map.next_key::()? { + key.check(what).map_err(de::Error::custom)?; match entries.entry(key) { Entry::Occupied(entry) => { return Err(de::Error::custom(format!( "duplicate {what} `{}`: contract map keys must be unique", - entry.key() + entry.key().shown() ))); } Entry::Vacant(entry) => { diff --git a/crates/runvil-internal/parser/src/contract/models.rs b/crates/runvil-internal/parser/src/contract/models.rs index 01b740ac9..98fe121d7 100644 --- a/crates/runvil-internal/parser/src/contract/models.rs +++ b/crates/runvil-internal/parser/src/contract/models.rs @@ -1,172 +1,76 @@ -//! Parses the `models` frontmatter key into declared model roles. +//! Parses the `models` frontmatter key into the requirements a prompt puts on +//! its model roles. //! -//! A role is a slot: a fill function at prepare maps it to a concrete model -//! and checks the hard keywords and the context minimum against the filled -//! model info. Parse only validates shape and exposes the declaration. - -use std::collections::{BTreeMap, btree_map}; +//! A prompt has five fixed roles, so the key is optional and only attaches +//! requirements to them: a minimum context window and a thinking switch. +//! The run's first step checks each requirement against the model that +//! fills the role. +//! Parse only validates shape: a role name outside the five, a field outside +//! the two, a repeated role, and a zero context minimum are errors. + +use std::collections::BTreeMap; use std::num::NonZeroU32; +use runvil_types::models::ModelRole; use serde::Deserialize; use crate::contract::{self, ContractKeys}; -/// A keyword that describes the model a role needs. -/// -/// Keywords are hard or soft. When a run is prepared, each role's hard -/// keywords (`thinking` and `no-thinking`) and its minimum context window -/// are checked against the model info of the model that fills the role. -/// Soft keywords document the author's intent. +/// The requirements a prompt puts on one model role. /// -/// The set of keywords is fixed, and a keyword outside the set is a parse -/// error. Adding a keyword is a change to the prompt language. -#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Deserialize)] -#[serde(rename_all = "kebab-case")] -#[non_exhaustive] -pub enum ModelKeyword { - /// A hard keyword: the model supports extended thinking. - Thinking, - /// A hard keyword: the model runs with thinking turned off. - NoThinking, - /// A soft keyword: a frontier-capability model. - Frontier, - /// A soft keyword: a fast model. - Fast, - /// A soft keyword: a small model. - Small, - /// A soft keyword: a creative model. - Creative, - /// A soft keyword: a chat-tuned model. - Chat, -} - -/// One model role a prompt declares: its keywords, minimum context window, -/// and description. -/// -/// A role is a slot for a model the prompt needs. When the run is -/// prepared, a concrete model fills the slot. -#[derive(Debug, Clone, PartialEq, Eq, Deserialize)] +/// `min_context` is the smallest context window, in tokens, the model that +/// fills the role may have. `thinking: true` requires a model that can think +/// and turns thinking on for the role's requests, and `thinking: false` +/// requires a model that can run without thinking and turns it off. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)] #[serde(deny_unknown_fields)] #[non_exhaustive] -pub struct ModelRole { - /// The declared keywords (closed vocabulary). - #[serde(default)] - keywords: Vec, +pub struct RoleRequirements { /// The minimum context window in tokens. #[serde(default)] min_context: Option, - /// The role's prose description. + /// The thinking requirement. #[serde(default)] - description: Option, + thinking: Option, } -impl ModelRole { - /// Returns the declared keywords. - #[must_use] - pub fn keywords(&self) -> &[ModelKeyword] { - &self.keywords - } - +impl RoleRequirements { /// Returns the minimum context window in tokens, when declared. #[must_use] pub fn min_context(&self) -> Option { self.min_context } - /// Returns the role's description, when declared. + /// Returns the thinking requirement, when declared: `Some(true)` for a + /// model that can think, `Some(false)` for one that can run without + /// thinking. #[must_use] - pub fn description(&self) -> Option<&str> { - self.description.as_deref() + pub fn thinking(&self) -> Option { + self.thinking } } -/// The model roles a prompt declares under the `models` frontmatter key, -/// keyed by label. -/// -/// A label is a name local to the prompt. It follows the same grammar as -/// an adhoc tool alias: `[A-Za-z][A-Za-z0-9_-]{0,63}`. -/// -/// A script reaches a role's model handle with `models.get(label)`. A label -/// is never a Lua global, so it may be a reserved name. +/// The requirements a prompt declares under the `models` frontmatter key, +/// keyed by the role they apply to. #[derive(Debug, Clone, Default, PartialEq, Eq)] -#[non_exhaustive] -pub struct ModelRoles { - roles: BTreeMap, -} - -impl ModelRoles { - /// Returns the role declared under `label`, when present. - #[must_use] - pub fn get(&self, label: &str) -> Option<&ModelRole> { - self.roles.get(label) - } - - /// Iterates the declared roles as `(label, role)` pairs, in label order. - pub fn iter(&self) -> ModelRolesIter<'_> { - ModelRolesIter { - roles: self.roles.iter(), - } - } - - /// Returns the number of declared roles. - #[must_use] - pub fn len(&self) -> usize { - self.roles.len() - } - - /// Returns whether the set of declared roles is empty. - #[must_use] - pub fn is_empty(&self) -> bool { - self.roles.is_empty() - } -} - -impl<'a> IntoIterator for &'a ModelRoles { - type Item = (&'a str, &'a ModelRole); - type IntoIter = ModelRolesIter<'a>; - - fn into_iter(self) -> ModelRolesIter<'a> { - self.iter() - } -} - -/// Yields the declared roles of one [`ModelRoles`] as `(label, role)` pairs, -/// in label order. -/// -/// [`ModelRoles::iter`] returns it. -#[derive(Debug, Clone)] -#[must_use = "an iterator does nothing until it is consumed"] -pub struct ModelRolesIter<'a> { - roles: btree_map::Iter<'a, String, ModelRole>, +pub(crate) struct RoleMap { + roles: BTreeMap, } -impl<'a> Iterator for ModelRolesIter<'a> { - type Item = (&'a str, &'a ModelRole); - - fn next(&mut self) -> Option<(&'a str, &'a ModelRole)> { - self.roles - .next() - .map(|(label, role)| (label.as_str(), role)) - } - - fn size_hint(&self) -> (usize, Option) { - self.roles.size_hint() +impl RoleMap { + /// Returns the requirements declared for `role`, when present. + pub(crate) fn get(&self, role: ModelRole) -> Option<&RoleRequirements> { + self.roles.get(&role) } } -impl ExactSizeIterator for ModelRolesIter<'_> {} - -impl<'de> Deserialize<'de> for ModelRoles { +impl<'de> Deserialize<'de> for RoleMap { fn deserialize(deserializer: D) -> std::result::Result where D: serde::Deserializer<'de>, { - let roles = contract::deserialize_contract_map( - deserializer, - ContractKeys { - what: "model role label", - }, - )?; - Ok(ModelRoles { roles }) + let roles = + contract::deserialize_contract_map(deserializer, ContractKeys { what: "model role" })?; + Ok(RoleMap { roles }) } } diff --git a/crates/runvil-internal/parser/src/contract/tests.rs b/crates/runvil-internal/parser/src/contract/tests.rs index 09b878d3f..4da448975 100644 --- a/crates/runvil-internal/parser/src/contract/tests.rs +++ b/crates/runvil-internal/parser/src/contract/tests.rs @@ -3,7 +3,9 @@ use std::num::NonZeroU32; -use super::{ArgType, ArgsDeclIter, ModelKeyword, ModelRolesIter}; +use runvil_types::models::ModelRole; + +use super::{ArgType, ArgsDeclIter}; use crate::ParseErrorKind; use crate::test_support::{assert_parse_error, parse_err, parse_ok, prompt}; @@ -17,9 +19,8 @@ fn the_full_contract_declaration_parses_and_round_trips() { uses: [web, mcp] args: { use_mcp: { type: boolean, default: true, \ description: Search MCP-connected private sources } } - models: { analyst: { keywords: [frontier, thinking], min_context: 200000, \ - description: deep reasoning }, \ - triage: { keywords: [fast, small], description: quick triage of search results } } + models: { strong: { min_context: 200000, thinking: true }, \ + fast: { thinking: false } } ", &[], )); @@ -40,19 +41,16 @@ fn the_full_contract_declaration_parses_and_round_trips() { Some("Search MCP-connected private sources") ); - let analyst = fm.models().get("analyst").expect("the role is declared"); - assert_eq!( - analyst.keywords(), - &[ModelKeyword::Frontier, ModelKeyword::Thinking] - ); - assert_eq!(analyst.min_context(), NonZeroU32::new(200_000)); - assert_eq!(analyst.description(), Some("deep reasoning")); - let triage = fm.models().get("triage").expect("the role is declared"); - assert_eq!( - triage.keywords(), - &[ModelKeyword::Fast, ModelKeyword::Small] - ); - assert_eq!(triage.min_context(), None); + let strong = fm + .role_requirements(ModelRole::Strong) + .expect("the role is declared"); + assert_eq!(strong.min_context(), NonZeroU32::new(200_000)); + assert_eq!(strong.thinking(), Some(true)); + let fast = fm + .role_requirements(ModelRole::Fast) + .expect("the role is declared"); + assert_eq!(fast.min_context(), None); + assert_eq!(fast.thinking(), Some(false)); } #[test] @@ -60,7 +58,12 @@ fn a_prompt_without_contract_keys_behaves_exactly_as_today() { let prompt = parse_ok(&prompt("", &[])); let fm = prompt.frontmatter(); assert!(fm.uses().is_empty()); - assert!(fm.models().is_empty()); + for role in ModelRole::ALL { + assert!( + fm.role_requirements(role).is_none(), + "a prompt with no models key declares nothing for {role:?}" + ); + } } #[test] @@ -85,24 +88,13 @@ fn an_unknown_key_inside_a_contract_entry_is_rejected() { // typo'd field is an authoring error, not silently ignored. for front in [ "args: { flag: { tipe: boolean } }", - "models: { analyst: { keyword: [fast] } }", + "models: { strong: { tier: 1 } }", ] { let error = parse_err(&prompt(front, &[])); assert_eq!(error.kind(), ParseErrorKind::Frontmatter, "{error}"); } } -#[test] -fn every_reserved_name_the_grammar_allows_parses_as_a_model_role_label() { - for (name, _) in runvil_lua::RESERVED_NAMES { - if name.starts_with('_') { - continue; - } - let prompt = parse_ok(&prompt(&format!("models: {{ '{name}': {{}} }}"), &[])); - assert!(prompt.frontmatter().models().get(name).is_some(), "{name}"); - } -} - #[test] fn an_arg_name_may_be_a_reserved_name_because_args_are_argv_fields() { let front = @@ -204,35 +196,44 @@ fn an_unknown_arg_type_is_rejected() { } #[test] -fn model_roles_round_trip_with_defaults() { - let prompt = parse_ok(&prompt( - "models: { analyst: { keywords: [no-thinking, creative, chat], min_context: 32000, \ - description: deep reasoning }, spare: {} }", - &[], - )); - let roles = prompt.frontmatter().models(); - assert_eq!(roles.len(), 2); - let analyst = roles.get("analyst").expect("declared"); - assert_eq!( - analyst.keywords(), - &[ - ModelKeyword::NoThinking, - ModelKeyword::Creative, - ModelKeyword::Chat - ] +fn each_role_declared_with_both_requirements_reads_them_back() { + for role in ModelRole::ALL { + let front = format!( + "models: {{ {}: {{ min_context: 4096, thinking: true }} }}", + role.as_str() + ); + let prompt = parse_ok(&prompt(&front, &[])); + let declared = prompt + .frontmatter() + .role_requirements(role) + .expect("the role is declared"); + assert_eq!(declared.min_context(), NonZeroU32::new(4096), "{role:?}"); + assert_eq!(declared.thinking(), Some(true), "{role:?}"); + } +} + +#[test] +fn an_empty_role_declares_neither_requirement() { + let prompt = parse_ok(&prompt("models: { fast: {} }", &[])); + let fast = prompt + .frontmatter() + .role_requirements(ModelRole::Fast) + .expect("an empty entry still declares the role"); + assert_eq!(fast.min_context(), None); + assert_eq!(fast.thinking(), None); + assert!( + prompt + .frontmatter() + .role_requirements(ModelRole::Strong) + .is_none(), + "a role the prompt does not name has no entry" ); - assert_eq!(analyst.min_context(), NonZeroU32::new(32_000)); - let spare = roles.get("spare").expect("declared"); - assert!(spare.keywords().is_empty()); - assert_eq!(spare.min_context(), None); - assert_eq!(spare.description(), None); } #[test] -fn args_and_model_roles_iterate_in_name_order_through_named_iterators() { +fn args_iterate_in_name_order_through_a_named_iterator() { let prompt = parse_ok(&prompt( - "args: { query: { type: string }, limit: { type: integer } } - models: { triage: {}, analyst: {} }", + "args: { query: { type: string }, limit: { type: integer } }", &[], )); let fm = prompt.frontmatter(); @@ -246,35 +247,38 @@ fn args_and_model_roles_iterate_in_name_order_through_named_iterators() { kinds.push(decl.kind()); } assert_eq!(kinds, [ArgType::Integer, ArgType::String]); +} + +#[test] +fn an_unknown_role_name_is_a_parse_error_that_lists_the_five_roles() { + assert_parse_error( + &prompt("models: { reviewer: {} }", &[]), + ParseErrorKind::Frontmatter, + "expected one of `default`, `strong`, `small`, `fast`, `creative`", + ); +} - let roles: ModelRolesIter<'_> = fm.models().iter(); - assert_eq!(roles.len(), 2); - let labels: Vec<&str> = roles.map(|(label, _)| label).collect(); - assert_eq!(labels, ["analyst", "triage"]); - let by_reference: Vec<&str> = fm.models().into_iter().map(|(label, _)| label).collect(); - assert_eq!(by_reference, labels); +#[test] +fn an_unknown_field_in_a_role_is_a_parse_error_that_names_the_two_fields() { + assert_parse_error( + &prompt("models: { strong: { tier: 1 } }", &[]), + ParseErrorKind::Frontmatter, + "unknown field `tier`, expected `min_context` or `thinking`", + ); } #[test] -fn an_unknown_model_keyword_is_a_parse_error() { - // The keyword vocabulary is closed; `multimodal` is a non-goal and a - // typo must fail at parse rather than being silently ignored. +fn a_duplicate_role_key_is_rejected() { let error = parse_err(&prompt( - "models: { analyst: { keywords: [multimodal] } }", + "models: { strong: { min_context: 1 }, strong: { thinking: true } }", &[], )); - assert_eq!(error.kind(), ParseErrorKind::Frontmatter); -} - -#[test] -fn model_role_labels_must_match_the_alias_grammar() { - let error = parse_err(&prompt("models: { '1analyst': { keywords: [fast] } }", &[])); - assert_eq!(error.kind(), ParseErrorKind::Frontmatter); + assert_eq!(error.kind(), ParseErrorKind::Frontmatter, "{error}"); } #[test] fn a_zero_min_context_is_rejected() { - let error = parse_err(&prompt("models: { analyst: { min_context: 0 } }", &[])); + let error = parse_err(&prompt("models: { strong: { min_context: 0 } }", &[])); assert_eq!(error.kind(), ParseErrorKind::Frontmatter); } diff --git a/crates/runvil-internal/parser/src/detail.rs b/crates/runvil-internal/parser/src/detail.rs index e0c165279..dafdcc210 100644 --- a/crates/runvil-internal/parser/src/detail.rs +++ b/crates/runvil-internal/parser/src/detail.rs @@ -5,8 +5,7 @@ //! crates reach it. use crate::{ - Block, Error, Frontmatter, MaxToolIterations, ModelKeyword, ParseError, ParseErrorKind, Prompt, - Section, + Block, Error, Frontmatter, MaxToolIterations, ParseError, ParseErrorKind, Prompt, Section, }; /// Unwraps a parse failure into the internal error it classifies, so the @@ -65,18 +64,3 @@ pub fn entry(prompt: &Prompt) -> Option<&Section> { pub fn max_tool_iterations(frontmatter: &Frontmatter) -> MaxToolIterations { frontmatter.max_tool_iterations } - -/// Returns the keyword's kebab-case spelling, the one the frontmatter -/// writes. -#[must_use] -pub fn keyword_name(keyword: ModelKeyword) -> &'static str { - match keyword { - ModelKeyword::Thinking => "thinking", - ModelKeyword::NoThinking => "no-thinking", - ModelKeyword::Frontier => "frontier", - ModelKeyword::Fast => "fast", - ModelKeyword::Small => "small", - ModelKeyword::Creative => "creative", - ModelKeyword::Chat => "chat", - } -} diff --git a/crates/runvil-internal/parser/src/frontmatter.rs b/crates/runvil-internal/parser/src/frontmatter.rs index 4aedd9de4..ec3927caa 100644 --- a/crates/runvil-internal/parser/src/frontmatter.rs +++ b/crates/runvil-internal/parser/src/frontmatter.rs @@ -2,9 +2,11 @@ use std::num::NonZeroU32; +use runvil_types::models::ModelRole; use runvil_types::packs::PackRef; -use crate::{ArgsDecl, Error, ModelRoles, ParseErrorKind, Result}; +use crate::contract::RoleMap; +use crate::{ArgsDecl, Error, ParseErrorKind, Result, RoleRequirements}; /// A declared input or output file in a prompt's frontmatter. /// @@ -75,9 +77,9 @@ pub struct Frontmatter { /// declaration (one optional string field named `prose`). #[serde(default)] args: ArgsDecl, - /// Declared model roles: label to keywords, minimum, and description. + /// The requirements declared per model role. #[serde(default)] - models: ModelRoles, + models: RoleMap, } /// The largest explicit `max_tool_iterations` a prompt may declare. @@ -208,10 +210,11 @@ impl Frontmatter { &self.args } - /// Returns the declared model roles, which map each label to a role. + /// Returns the requirements the prompt declares for `role`, or `None` + /// when it does not name the role. #[must_use] - pub fn models(&self) -> &ModelRoles { - &self.models + pub fn role_requirements(&self, role: ModelRole) -> Option<&RoleRequirements> { + self.models.get(role) } } diff --git a/crates/runvil-internal/parser/src/lib.rs b/crates/runvil-internal/parser/src/lib.rs index 4e958bad3..c905d2caf 100644 --- a/crates/runvil-internal/parser/src/lib.rs +++ b/crates/runvil-internal/parser/src/lib.rs @@ -39,9 +39,7 @@ mod sections; pub mod test_support; pub use body_text::BodyText; -pub use contract::{ - ArgDecl, ArgType, ArgsDecl, ArgsDeclIter, ModelKeyword, ModelRole, ModelRoles, ModelRolesIter, -}; +pub use contract::{ArgDecl, ArgType, ArgsDecl, ArgsDeclIter, RoleRequirements}; use error::Result; pub use error::{Error, ParseError, ParseErrorKind}; #[cfg(test)] diff --git a/crates/runvil-internal/parser/src/parse.rs b/crates/runvil-internal/parser/src/parse.rs index 4d4398d46..c4bceddf5 100644 --- a/crates/runvil-internal/parser/src/parse.rs +++ b/crates/runvil-internal/parser/src/parse.rs @@ -3,7 +3,7 @@ use std::num::NonZeroU32; use std::sync::Arc; -use runvil_types::emitter::{DebugMode, Emitter, EventSink}; +use runvil_types::emitter::{Emitter, EventSink}; use runvil_types::event::{Event, lifecycle}; use crate::contract::{self, PackListError}; @@ -60,7 +60,7 @@ impl Prompt { execution: &str, ) -> (std::result::Result, Vec) { let sink = EventSink::default(); - let emitter = Emitter::root(sink.clone(), execution, DebugMode::Off); + let emitter = Emitter::root(sink.clone(), execution, false); emitter.report("Prompt", lifecycle::PARSE_STARTED); let result = Self::parse_inner(input, &emitter); emitter.report( diff --git a/crates/runvil-internal/types/src/cancel-tests.rs b/crates/runvil-internal/types/src/cancel-tests.rs index c4ceb4f82..f8a44c836 100644 --- a/crates/runvil-internal/types/src/cancel-tests.rs +++ b/crates/runvil-internal/types/src/cancel-tests.rs @@ -34,7 +34,7 @@ impl Counter { /// Compile-time proof that a handle can cross thread boundaries and live for /// the whole program: a caller may move clones onto other threads, and the -/// Engine keeps one in `RunContext` while `Run` itself is `Send`. +/// Engine keeps one in `RunConfig` while `Run` itself is `Send`. const _: () = assert_send_sync::(); #[test] diff --git a/crates/runvil-internal/types/src/cancel.rs b/crates/runvil-internal/types/src/cancel.rs index 88dd37ae5..ca5aa4599 100644 --- a/crates/runvil-internal/types/src/cancel.rs +++ b/crates/runvil-internal/types/src/cancel.rs @@ -7,7 +7,7 @@ //! reaches every task while one task can be cancelled without touching its //! siblings or its owner. //! -//! This is the handle the Engine's `RunContext` holds. A caller that must +//! This is the handle the Engine's `RunConfig` holds. A caller that must //! wait on the flag itself awaits [`CancelHandle::cancelled`], a std-only //! future the cancel itself wakes, in place of a timer that polls the flag. diff --git a/crates/runvil-internal/types/src/emitter-tests.rs b/crates/runvil-internal/types/src/emitter-tests.rs index 52f072475..065dabb23 100644 --- a/crates/runvil-internal/types/src/emitter-tests.rs +++ b/crates/runvil-internal/types/src/emitter-tests.rs @@ -2,7 +2,7 @@ use std::sync::Arc; -use super::{DebugMode, Emitter, EventSink, LibraryLoad}; +use super::{Emitter, EventSink, LibraryLoad}; use crate::event::{Event, lifecycle}; use crate::ids::{AbandonReason, ChainId, Provenance, RoundId, TaskId}; use crate::tools::{OutputTrust, ToolId}; @@ -12,7 +12,7 @@ fn root() -> TaskId { } fn emitter(sink: &EventSink, task: TaskId) -> Emitter { - Emitter::new(sink.clone(), task, Arc::from("run-1"), DebugMode::Off) + Emitter::new(sink.clone(), task, Arc::from("run-1"), false) } #[test] @@ -241,7 +241,7 @@ fn an_untrusted_tool_result_reports_the_event_with_trusted_false() { #[test] fn the_root_emitter_reports_under_task_zero_with_its_execution() { let sink = EventSink::default(); - let emitter = Emitter::root(sink.clone(), "parse-1", DebugMode::On); + let emitter = Emitter::root(sink.clone(), "parse-1", true); assert!(emitter.captures_debug()); assert_eq!(emitter.execution(), "parse-1"); assert_eq!(emitter.task(), &root()); diff --git a/crates/runvil-internal/types/src/emitter.rs b/crates/runvil-internal/types/src/emitter.rs index 651501de4..25f4b4cef 100644 --- a/crates/runvil-internal/types/src/emitter.rs +++ b/crates/runvil-internal/types/src/emitter.rs @@ -37,23 +37,6 @@ use crate::tools::{OutputTrust, ToolId}; #[path = "emitter-tests.rs"] mod tests; -/// Selects whether a run reports each model round's raw bodies as events. -/// -/// The bodies are the round's raw request and response, reported as -/// `Request` and `Response` events. Off by default. The `Chat` effect and -/// its answer already carry the bodies, so a caller that logs effects has -/// them. A caller that also wants them in the event stream selects `On`. -#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] -#[non_exhaustive] -pub enum DebugMode { - /// The model rounds skip the `Request` and `Response` events and - /// the body clones those events need. - #[default] - Off, - /// Every model round emits its raw request and response bodies. - On, -} - /// Names one phase of loading a library section into a section's environment. /// /// [`Emitter::library_load`] reports it. @@ -163,15 +146,16 @@ pub struct Emitter { /// The caller-chosen run identifier stamped on every event. execution: Arc, /// Whether the caller asked for raw request/response capture: the model - /// rounds emit `Request` and `Response` only when [`DebugMode::On`], - /// so a run that did not opt in never clones a body. - debug: DebugMode, + /// rounds emit `Request` and `Response` only when it is set, so a run + /// that did not opt in never clones a body. + debug: bool, } impl Emitter { - /// Builds the emitter for `task` over `sink`. + /// Builds the emitter for `task` over `sink`, capturing raw model-turn + /// bodies when `debug` is set. #[must_use] - pub fn new(sink: EventSink, task: TaskId, execution: Arc, debug: DebugMode) -> Self { + pub fn new(sink: EventSink, task: TaskId, execution: Arc, debug: bool) -> Self { Self { sink, task, @@ -183,7 +167,7 @@ impl Emitter { /// The root task's emitter over `sink`: the main walk is task `0`, /// and so is a prompt's parse, which happens before any run exists. #[must_use] - pub fn root(sink: EventSink, execution: &str, debug: DebugMode) -> Self { + pub fn root(sink: EventSink, execution: &str, debug: bool) -> Self { Self::new( sink, TaskId::from(ChainId::root()), @@ -220,7 +204,7 @@ impl Emitter { /// Whether the run captures raw model-turn bodies. #[must_use] pub fn captures_debug(&self) -> bool { - self.debug == DebugMode::On + self.debug } /// Stamps one issued effect: this task's next provenance, drawn from diff --git a/crates/runvil-internal/types/src/event-lifecycle.rs b/crates/runvil-internal/types/src/event-lifecycle.rs index 4d9adad73..b350c0de2 100644 --- a/crates/runvil-internal/types/src/event-lifecycle.rs +++ b/crates/runvil-internal/types/src/event-lifecycle.rs @@ -58,17 +58,11 @@ lifecycle_constants! { LUA_CHUNK_STARTED => LuaChunkStarted, LUA_CHUNK_SUCCEEDED => LuaChunkSucceeded, LUA_CHUNK_FAILED => LuaChunkFailed, - LUA_REPLY_BINDING_STARTED => LuaReplyBindingStarted, - LUA_REPLY_BINDING_SUCCEEDED => LuaReplyBindingSucceeded, - LUA_REPLY_BINDING_FAILED => LuaReplyBindingFailed, LUA_TEARDOWN_STARTED => LuaTeardownStarted, LUA_TEARDOWN_SUCCEEDED => LuaTeardownSucceeded, TOOL_SCOPE_VALIDATION_STARTED => ToolScopeValidationStarted, TOOL_SCOPE_VALIDATION_SUCCEEDED => ToolScopeValidationSucceeded, TOOL_SCOPE_VALIDATION_FAILED => ToolScopeValidationFailed, - MODEL_CATALOG_VALIDATION_STARTED => ModelCatalogValidationStarted, - MODEL_CATALOG_VALIDATION_SUCCEEDED => ModelCatalogValidationSucceeded, - MODEL_CATALOG_VALIDATION_FAILED => ModelCatalogValidationFailed, VFS_WRITE_SUCCEEDED => VfsWriteSucceeded, VFS_WRITE_FAILED => VfsWriteFailed, VFS_APPEND_SUCCEEDED => VfsAppendSucceeded, diff --git a/crates/runvil-internal/types/src/event-tests.rs b/crates/runvil-internal/types/src/event-tests.rs index 8bd6dac25..eaf810dc6 100644 --- a/crates/runvil-internal/types/src/event-tests.rs +++ b/crates/runvil-internal/types/src/event-tests.rs @@ -64,9 +64,6 @@ fn lifecycle_of(event: &Event) -> Option { Event::LuaChunkStarted { .. } => Some(lifecycle::LUA_CHUNK_STARTED), Event::LuaChunkSucceeded { .. } => Some(lifecycle::LUA_CHUNK_SUCCEEDED), Event::LuaChunkFailed { .. } => Some(lifecycle::LUA_CHUNK_FAILED), - Event::LuaReplyBindingStarted { .. } => Some(lifecycle::LUA_REPLY_BINDING_STARTED), - Event::LuaReplyBindingSucceeded { .. } => Some(lifecycle::LUA_REPLY_BINDING_SUCCEEDED), - Event::LuaReplyBindingFailed { .. } => Some(lifecycle::LUA_REPLY_BINDING_FAILED), Event::LuaTeardownStarted { .. } => Some(lifecycle::LUA_TEARDOWN_STARTED), Event::LuaTeardownSucceeded { .. } => Some(lifecycle::LUA_TEARDOWN_SUCCEEDED), Event::ToolScopeValidationStarted { .. } => Some(lifecycle::TOOL_SCOPE_VALIDATION_STARTED), @@ -74,15 +71,6 @@ fn lifecycle_of(event: &Event) -> Option { Some(lifecycle::TOOL_SCOPE_VALIDATION_SUCCEEDED) } Event::ToolScopeValidationFailed { .. } => Some(lifecycle::TOOL_SCOPE_VALIDATION_FAILED), - Event::ModelCatalogValidationStarted { .. } => { - Some(lifecycle::MODEL_CATALOG_VALIDATION_STARTED) - } - Event::ModelCatalogValidationSucceeded { .. } => { - Some(lifecycle::MODEL_CATALOG_VALIDATION_SUCCEEDED) - } - Event::ModelCatalogValidationFailed { .. } => { - Some(lifecycle::MODEL_CATALOG_VALIDATION_FAILED) - } Event::VfsWriteSucceeded { .. } => Some(lifecycle::VFS_WRITE_SUCCEEDED), Event::VfsWriteFailed { .. } => Some(lifecycle::VFS_WRITE_FAILED), Event::VfsAppendSucceeded { .. } => Some(lifecycle::VFS_APPEND_SUCCEEDED), @@ -109,12 +97,10 @@ fn lifecycle_of(event: &Event) -> Option { | Event::TaskFailed { .. } | Event::TaskCancelled { .. } | Event::TaskAbandoned { .. } - | Event::TaskResumed { .. } | Event::Thinking { .. } | Event::AssistantReply { .. } | Event::AssistantToolCalls { .. } | Event::ToolResult { .. } - | Event::TaskNote { .. } | Event::Request { .. } | Event::Response { .. } => None, } @@ -187,12 +173,6 @@ fn task_samples() -> Vec { task: task("0.0"), reason: AbandonReason::ToolLoopExhausted, }, - Event::TaskResumed { - execution: "run-1".to_owned(), - section: "Worker".to_owned(), - provenance: provenance("0.3", 0), - task: task("0.3"), - }, ] } @@ -246,13 +226,6 @@ fn content_samples() -> Vec { content: "file contents".to_owned(), trusted: false, }, - Event::TaskNote { - execution: "run-1".to_owned(), - section: "Worker".to_owned(), - provenance: provenance("0.1", 2), - task: task("0.1"), - text: "halfway".to_owned(), - }, ] } @@ -388,16 +361,15 @@ fn a_serialized_event_is_tagged_by_kind_with_its_coordinates_beside_the_payload( &event, r#"{"kind":"vfs_write_succeeded","execution":"run-1","section":"Gather","provenance":{"task":"0.2","seq":9}}"#, ); - let note = Event::TaskNote { + let succeeded = Event::TaskSucceeded { execution: "run-1".to_owned(), section: "Worker".to_owned(), provenance: provenance("0.1", 10), task: task("0.1"), - text: "halfway".to_owned(), }; assert_wire( - ¬e, - r#"{"kind":"task_note","execution":"run-1","section":"Worker","provenance":{"task":"0.1","seq":10},"task":"0.1","text":"halfway"}"#, + &succeeded, + r#"{"kind":"task_succeeded","execution":"run-1","section":"Worker","provenance":{"task":"0.1","seq":10},"task":"0.1"}"#, ); } @@ -443,7 +415,7 @@ fn every_event_exposes_its_coordinates() { provenance: provenance("0", 12), message: "hello".to_owned(), }, - Event::TaskResumed { + Event::TaskSucceeded { execution: "run-1".to_owned(), section: "Worker".to_owned(), provenance: provenance("0.1", 0), diff --git a/crates/runvil-internal/types/src/event.rs b/crates/runvil-internal/types/src/event.rs index ea91d965f..fb7ee4f3f 100644 --- a/crates/runvil-internal/types/src/event.rs +++ b/crates/runvil-internal/types/src/event.rs @@ -163,7 +163,7 @@ events! { ParseSucceeded {}, /// Prompt parsing or parse-time compilation returned an error. ParseFailed {}, - /// A run passed its version gate and began. + /// A run passed its start checks and began. RunStarted {}, /// A run returned a value. RunSucceeded {}, @@ -227,12 +227,6 @@ events! { LuaChunkSucceeded {}, /// A section VM failed to execute a Lua chunk. LuaChunkFailed {}, - /// A section VM began binding a model reply. - LuaReplyBindingStarted {}, - /// A section VM bound a model reply successfully. - LuaReplyBindingSucceeded {}, - /// A section VM failed to bind a model reply. - LuaReplyBindingFailed {}, /// A section VM began teardown. LuaTeardownStarted {}, /// A section VM completed teardown. @@ -244,15 +238,6 @@ events! { ToolScopeValidationSucceeded {}, /// A model-visible tool scope failed semantic validation. ToolScopeValidationFailed {}, - /// Validation of model bindings against the live model catalog - /// began. - ModelCatalogValidationStarted {}, - /// Validation of model bindings against the live model catalog - /// succeeded. - ModelCatalogValidationSucceeded {}, - /// Validation of model bindings against the live model catalog - /// failed. - ModelCatalogValidationFailed {}, // Lifecycle: operations on the run's store view (the `store.*` calls). /// A `store.write` call succeeded. VfsWriteSucceeded {}, @@ -341,12 +326,6 @@ events! { /// How the owner ended. reason: AbandonReason, }, - /// Reports an existing task revived from its record. The Engine - /// never emits it. - TaskResumed { - /// The task's id. - task: TaskId, - }, // Content. /// One completed block of model thinking. Thinking { @@ -417,16 +396,6 @@ events! { /// not nonce-wrapped). trusted: bool, }, - /// Reports a task's own progress note, set through `tasks.note`. - /// The Engine never emits it: it stores the note on the task's - /// chain, and the task's owner reads it through `tasks.status`. - TaskNote { - /// The task that set the note. - task: TaskId, - /// The note: untrusted, authored by the task's model or its - /// Lua. - text: String, - }, // Debug. /// The JSON body sent to the chat-completions endpoint for one /// model turn: raw, unredacted, and including the full prompt. diff --git a/crates/runvil-internal/types/src/ids-tests.rs b/crates/runvil-internal/types/src/ids-tests.rs index 39c676f5b..a51e3557d 100644 --- a/crates/runvil-internal/types/src/ids-tests.rs +++ b/crates/runvil-internal/types/src/ids-tests.rs @@ -48,7 +48,12 @@ fn malformed_paths_are_rejected() { let error = input .parse::() .expect_err("a malformed path is rejected"); - assert_eq!(error.input(), input); + assert_eq!( + error.to_string(), + format!( + "invalid chain id `{input}`: required a dot-separated path of decimal components" + ) + ); } } diff --git a/crates/runvil-internal/types/src/ids.rs b/crates/runvil-internal/types/src/ids.rs index fe3f39096..85748ef96 100644 --- a/crates/runvil-internal/types/src/ids.rs +++ b/crates/runvil-internal/types/src/ids.rs @@ -29,6 +29,8 @@ use std::str::FromStr; use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use crate::names::IdError; + #[cfg(test)] #[path = "ids-tests.rs"] mod tests; @@ -81,43 +83,16 @@ impl fmt::Display for ChainId { } } -/// The error returned when text fails to parse as a [`ChainId`] or -/// [`TaskId`] path. -/// -/// A valid path is one or more components joined by dots, such as `0.2.0`. -/// Each component is plain decimal digits that fit in a `u32`. -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct ParseIdError { - /// The rejected text. - input: String, -} - -impl ParseIdError { - /// Returns the rejected text. - #[must_use] - pub fn input(&self) -> &str { - &self.input - } -} - -impl fmt::Display for ParseIdError { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!( - f, - "invalid chain id `{}`: required a dot-separated path of decimal components", - self.input - ) - } -} - -impl std::error::Error for ParseIdError {} - impl FromStr for ChainId { - type Err = ParseIdError; + type Err = IdError; + /// Parses a path: one or more components joined by dots, such as + /// `0.2.0`, each plain decimal digits that fit in a `u32`. fn from_str(input: &str) -> Result { - let reject = || ParseIdError { - input: input.to_owned(), + let reject = || { + IdError::new(format!( + "invalid chain id `{input}`: required a dot-separated path of decimal components" + )) }; if input.is_empty() { return Err(reject()); @@ -133,7 +108,7 @@ impl FromStr for ChainId { } component.parse::().map_err(|_| reject()) }) - .collect::, ParseIdError>>() + .collect::, IdError>>() .map(Self) } } @@ -173,7 +148,7 @@ impl fmt::Display for TaskId { } impl FromStr for TaskId { - type Err = ParseIdError; + type Err = IdError; fn from_str(input: &str) -> Result { input.parse().map(Self) diff --git a/crates/runvil-internal/types/src/lib.rs b/crates/runvil-internal/types/src/lib.rs index 0146039a9..91e5a562a 100644 --- a/crates/runvil-internal/types/src/lib.rs +++ b/crates/runvil-internal/types/src/lib.rs @@ -20,12 +20,10 @@ //! holds an implementation. [`ids`] is the hierarchical, deterministic //! identity of a run's chains and tasks and the [`Provenance`](ids::Provenance) //! replay key stamped on every effect and event; [`timestamp`] is the UTC -//! instant a run starts from, rendered over std alone; [`seed`] is the +//! instant a run starts from, rendered over std alone; and [`seed`] is the //! caller's [`RunSeed`](seed::RunSeed) and the one-way derivations from -//! it, the run id [`RunUid`](seed::RunUid) among them; and [`replay`] holds -//! the behavior [`Flags`](replay::Flags) a run records. This crate depends -//! on no other runvil crate, so every runvil crate may depend on -//! it, and it declares no async runtime. +//! it. This crate depends on no other runvil crate, so every runvil crate +//! may depend on it, and it declares no async runtime. //! //! ## Invariants //! @@ -43,7 +41,6 @@ pub mod metrics; pub mod models; pub mod names; pub mod packs; -pub mod replay; pub mod seed; #[cfg(test)] mod test_support; diff --git a/crates/runvil-internal/types/src/models.rs b/crates/runvil-internal/types/src/models.rs index 336f29cf0..c76abc14b 100644 --- a/crates/runvil-internal/types/src/models.rs +++ b/crates/runvil-internal/types/src/models.rs @@ -9,6 +9,8 @@ use std::num::NonZeroU32; use serde::Deserialize; +use crate::names::IdError; + /// The stable identity of one catalogued model. /// /// An identity pairs a server namespace with a model name. A model the @@ -33,12 +35,12 @@ impl ModelId { /// Builds an identity from its server namespace and model name. /// /// # Errors - /// Returns [`ModelIdError`] if `server` or `name` is empty or contains a + /// Returns [`IdError`] if `server` or `name` is empty or contains a /// control character, so every `ModelId` is usable. pub fn new( server: impl Into, name: impl Into, - ) -> std::result::Result { + ) -> std::result::Result { let server = server.into(); let name = name.into(); Self::validate("server", &server)?; @@ -51,9 +53,9 @@ impl ModelId { /// `name` is the name the gateway lists the model under. /// /// # Errors - /// Returns [`ModelIdError`] if `name` is empty or contains a control + /// Returns [`IdError`] if `name` is empty or contains a control /// character. - pub fn gateway(name: impl Into) -> std::result::Result { + pub fn gateway(name: impl Into) -> std::result::Result { Self::new(Self::GATEWAY, name) } @@ -62,20 +64,15 @@ impl ModelId { /// Rejection is by Unicode scalar, not raw byte: every control /// character is refused, including C1 controls such as U+0085 (NEL) whose /// UTF-8 encoding a byte-range scan would miss. - fn validate(field: &'static str, value: &str) -> std::result::Result<(), ModelIdError> { - if value.is_empty() { - return Err(ModelIdError { - field, - reason: "must not be empty", - }); - } - if value.chars().any(char::is_control) { - return Err(ModelIdError { - field, - reason: "must not contain a control character", - }); - } - Ok(()) + fn validate(field: &'static str, value: &str) -> std::result::Result<(), IdError> { + let reason = if value.is_empty() { + "must not be empty" + } else if value.chars().any(char::is_control) { + "must not contain a control character" + } else { + return Ok(()); + }; + Err(IdError::new(format!("invalid model id: {field} {reason}"))) } /// Returns the server namespace. @@ -91,51 +88,6 @@ impl ModelId { } } -/// The reason building a [`ModelId`] from its components failed. -#[derive(Debug, Clone, PartialEq, Eq)] -#[non_exhaustive] -pub struct ModelIdError { - /// Which component was rejected (`server` or `name`). - field: &'static str, - /// Why it was rejected. - reason: &'static str, -} - -impl std::fmt::Display for ModelIdError { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(f, "invalid model id: {} {}", self.field, self.reason) - } -} - -impl std::error::Error for ModelIdError {} - -/// The reason building a [`ModelCatalog`] from its `ModelInfo` values failed. -#[derive(Debug, Clone, PartialEq, Eq)] -#[non_exhaustive] -pub enum ModelCatalogError { - /// Two `ModelInfo` values shared one stable [`ModelId`], which would make - /// lookups ambiguous. - #[non_exhaustive] - DuplicateId { - /// The repeated identity's server namespace. - server: String, - /// The repeated identity's model name. - name: String, - }, -} - -impl std::fmt::Display for ModelCatalogError { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - ModelCatalogError::DuplicateId { server, name } => { - write!(f, "duplicate model identity in catalog: {server}/{name}") - } - } - } -} - -impl std::error::Error for ModelCatalogError {} - /// Whether a catalogued model can emit thinking tokens. #[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)] #[serde(rename_all = "lowercase")] @@ -149,6 +101,50 @@ pub enum ThinkingMode { Switchable, } +/// One of the five fixed model tiers a prompt can ask for. +/// +/// Each role is a tier the caller maps to a model. A role with no model of its +/// own uses the model mapped to [`ModelRole::Default`]. The names are +/// lowercase on the wire, and [`ModelRole::ALL`] lists them in a fixed order. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, Deserialize)] +#[serde(rename_all = "kebab-case")] +#[non_exhaustive] +pub enum ModelRole { + /// The model every other role falls back to. + Default, + /// The most capable tier, for hard reasoning. + Strong, + /// A compact tier, for light work. + Small, + /// The lowest-latency tier. + Fast, + /// The tier for open-ended writing. + Creative, +} + +impl ModelRole { + /// Every role, in declaration order. + pub const ALL: [ModelRole; 5] = [ + ModelRole::Default, + ModelRole::Strong, + ModelRole::Small, + ModelRole::Fast, + ModelRole::Creative, + ]; + + /// Returns the role's lowercase name, the text a prompt writes. + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + ModelRole::Default => "default", + ModelRole::Strong => "strong", + ModelRole::Small => "small", + ModelRole::Fast => "fast", + ModelRole::Creative => "creative", + } + } +} + /// Describes one catalogued model. /// /// It holds the model's identity, description, context window, thinking @@ -242,19 +238,20 @@ impl ModelCatalog { /// given in. /// /// # Errors - /// Returns [`ModelCatalogError::DuplicateId`] when two `ModelInfo` values - /// share one stable [`ModelId`], so each identity in the catalog names - /// exactly one model info. + /// Returns [`IdError`] when two `ModelInfo` values share one stable + /// [`ModelId`], so each identity in the catalog names exactly one model + /// info. pub fn new( models: impl IntoIterator, - ) -> std::result::Result { + ) -> std::result::Result { let models: Vec = models.into_iter().collect(); for (index, model) in models.iter().enumerate() { if models[..index].iter().any(|prior| prior.id() == model.id()) { - return Err(ModelCatalogError::DuplicateId { - server: model.id().server().to_owned(), - name: model.id().name().to_owned(), - }); + return Err(IdError::new(format!( + "duplicate model identity in catalog: {}/{}", + model.id().server(), + model.id().name() + ))); } } Ok(Self { models }) @@ -305,10 +302,27 @@ mod tests { #[test] fn model_id_rejects_empty_and_control_characters() { - assert!(ModelId::gateway("").is_err()); - assert!(ModelId::new("", "name").is_err()); - assert!(ModelId::new("server", "").is_err()); - assert!(ModelId::new("server", "na\nme").is_err()); + let message = |result: std::result::Result| { + result + .expect_err("the model id must be rejected") + .to_string() + }; + assert_eq!( + message(ModelId::gateway("")), + "invalid model id: name must not be empty" + ); + assert_eq!( + message(ModelId::new("", "name")), + "invalid model id: server must not be empty" + ); + assert_eq!( + message(ModelId::new("server", "")), + "invalid model id: name must not be empty" + ); + assert_eq!( + message(ModelId::new("server", "na\nme")), + "invalid model id: name must not contain a control character" + ); assert!(ModelId::gateway("valid-alias").is_ok()); } @@ -367,7 +381,67 @@ mod tests { }; let err = ModelCatalog::new([info("dup"), info("dup")]) .expect_err("a catalog with duplicate ids must be rejected"); - assert!(matches!(err, ModelCatalogError::DuplicateId { .. })); + assert_eq!( + err.to_string(), + "duplicate model identity in catalog: gateway/dup" + ); assert!(ModelCatalog::new([info("a"), info("b")]).is_ok()); } + + #[test] + fn each_model_role_name_deserializes_to_its_variant() { + let cases = [ + ("default", ModelRole::Default), + ("strong", ModelRole::Strong), + ("small", ModelRole::Small), + ("fast", ModelRole::Fast), + ("creative", ModelRole::Creative), + ]; + for (name, expected) in cases { + let role: ModelRole = serde_json::from_str(&format!("\"{name}\"")) + .expect("a model role name deserializes"); + assert_eq!(role, expected, "model_role name {name}"); + } + } + + #[test] + fn model_role_all_lists_the_five_roles_in_order_and_round_trips_through_as_str() { + assert_eq!( + ModelRole::ALL, + [ + ModelRole::Default, + ModelRole::Strong, + ModelRole::Small, + ModelRole::Fast, + ModelRole::Creative, + ] + ); + for role in ModelRole::ALL { + let back: ModelRole = serde_json::from_str(&format!("\"{}\"", role.as_str())) + .expect("a model_role as_str text deserializes"); + assert_eq!(back, role); + } + } + + #[test] + fn an_unknown_model_role_name_fails_and_the_error_names_all_five_roles() { + let err = serde_json::from_str::("\"reviewer\"") + .expect_err("an unknown model_role name must be rejected") + .to_string(); + for name in ["default", "strong", "small", "fast", "creative"] { + assert!(err.contains(name), "model_role error {err:?} omits {name}"); + } + } + + #[test] + fn a_model_role_works_as_a_btree_map_key_ordered_like_all() { + let map: std::collections::BTreeMap = ModelRole::ALL + .iter() + .rev() + .enumerate() + .map(|(index, role)| (*role, index)) + .collect(); + let keys: Vec = map.keys().copied().collect(); + assert_eq!(keys, ModelRole::ALL, "model_role keys follow the ALL order"); + } } diff --git a/crates/runvil-internal/types/src/names-tests.rs b/crates/runvil-internal/types/src/names-tests.rs index 5180d12c6..8b3ce5387 100644 --- a/crates/runvil-internal/types/src/names-tests.rs +++ b/crates/runvil-internal/types/src/names-tests.rs @@ -1,11 +1,16 @@ -//! Tests for `GlobalName` parsing and its rejection kinds. +//! Tests for `GlobalName` parsing and its rejection messages. -use super::{GlobalName, GlobalNameErrorKind}; +use super::GlobalName; -fn kind_of(input: &str) -> GlobalNameErrorKind { +const EMPTY: &str = "invalid global name: segments must not be empty"; +const CONTROL: &str = "invalid global name: segments must not contain a control character"; +const CHARSET: &str = + "invalid global name: segments may contain only lowercase ASCII letters, digits, '-', '_', '.'"; + +fn message_of(input: &str) -> String { GlobalName::parse(input) .expect_err("the input must be rejected") - .kind() + .to_string() } #[test] @@ -53,44 +58,44 @@ fn dashes_underscores_and_dots_are_legal_segment_characters() { #[test] fn an_empty_string_is_rejected_as_an_empty_error() { - assert_eq!(kind_of(""), GlobalNameErrorKind::Empty); + assert_eq!(message_of(""), EMPTY); } #[test] fn an_empty_middle_segment_is_rejected_as_an_empty_error() { - assert_eq!(kind_of("runvil//web"), GlobalNameErrorKind::Empty); + assert_eq!(message_of("runvil//web"), EMPTY); } #[test] fn a_leading_separator_is_rejected_as_an_empty_error() { - assert_eq!(kind_of("/web"), GlobalNameErrorKind::Empty); + assert_eq!(message_of("/web"), EMPTY); } #[test] fn a_trailing_separator_is_rejected_as_an_empty_error() { - assert_eq!(kind_of("web/"), GlobalNameErrorKind::Empty); + assert_eq!(message_of("web/"), EMPTY); } #[test] fn uppercase_is_rejected_as_a_charset_error_because_comparison_is_case_sensitive() { - assert_eq!(kind_of("Web"), GlobalNameErrorKind::Charset); - assert_eq!(kind_of("web/Fetch"), GlobalNameErrorKind::Charset); + assert_eq!(message_of("Web"), CHARSET); + assert_eq!(message_of("web/Fetch"), CHARSET); } #[test] fn an_at_sign_is_rejected_as_a_charset_error_because_v1_is_unversioned() { - assert_eq!(kind_of("web@2"), GlobalNameErrorKind::Charset); + assert_eq!(message_of("web@2"), CHARSET); } #[test] fn a_control_character_is_rejected_as_a_control_error() { - assert_eq!(kind_of("we\tb"), GlobalNameErrorKind::Control); - assert_eq!(kind_of("web\u{7f}"), GlobalNameErrorKind::Control); + assert_eq!(message_of("we\tb"), CONTROL); + assert_eq!(message_of("web\u{7f}"), CONTROL); } #[test] fn non_ascii_is_rejected_as_a_charset_error() { - assert_eq!(kind_of("w\u{e9}b"), GlobalNameErrorKind::Charset); + assert_eq!(message_of("w\u{e9}b"), CHARSET); } #[test] @@ -103,5 +108,9 @@ fn from_str_parses_like_parse() { let error = "Web" .parse::() .expect_err("uppercase is rejected"); - assert_eq!(error.kind(), GlobalNameErrorKind::Charset); + assert_eq!( + error, + GlobalName::parse("Web").expect_err("uppercase is rejected") + ); + assert_eq!(error.to_string(), CHARSET); } diff --git a/crates/runvil-internal/types/src/names.rs b/crates/runvil-internal/types/src/names.rs index 2c74b64b7..41ba0ac3d 100644 --- a/crates/runvil-internal/types/src/names.rs +++ b/crates/runvil-internal/types/src/names.rs @@ -7,6 +7,9 @@ //! two or more, its Pack's id first (`web/fetch`). Segments are lowercase ASCII //! alphanumeric plus `-`, `_`, `.`, and comparison is case-sensitive. A //! name carries no version, so a `@` is a parse error. +//! +//! [`IdError`] is the error every id type and every catalog of ids returns, +//! whether the id is built on a name or not. use std::fmt; use std::str::FromStr; @@ -40,12 +43,20 @@ impl GlobalName { /// /// # Errors /// - /// Returns [`GlobalNameError`] when: + /// Returns [`IdError`] when: /// /// - a segment is empty, including the whole string being empty; /// - a segment contains a control byte; /// - a segment contains any other character outside the set. - pub fn parse(s: &str) -> Result { + pub fn parse(s: &str) -> Result { + GlobalName::validate(s) + .map_err(|reason| IdError::new(format!("invalid global name: {reason}"))) + } + + /// Parses `s` as [`GlobalName::parse`] does, but returns only the + /// reason it is invalid, so each id type built on a name can name + /// itself in its own error. + pub(crate) fn validate(s: &str) -> Result { let segments: Vec<&str> = s.split('/').collect(); for segment in &segments { validate_segment(segment)?; @@ -82,9 +93,9 @@ impl GlobalName { } impl FromStr for GlobalName { - type Err = GlobalNameError; + type Err = IdError; - fn from_str(s: &str) -> Result { + fn from_str(s: &str) -> Result { GlobalName::parse(s) } } @@ -104,74 +115,57 @@ impl fmt::Display for GlobalName { /// Validates one segment against the charset rule. /// /// A segment must be non-empty and contain only lowercase ASCII -/// alphanumeric characters plus `-`, `_`, `.`. A control byte is a -/// `Control` error; anything else outside the set - including uppercase -/// (comparison is case-sensitive), `@`, and non-ASCII - is a `Charset` -/// error. -fn validate_segment(segment: &str) -> Result<(), GlobalNameError> { +/// alphanumeric characters plus `-`, `_`, `.`. A control byte has its own +/// reason; anything else outside the set - including uppercase +/// (comparison is case-sensitive), `@`, and non-ASCII - shares the +/// charset reason. +fn validate_segment(segment: &str) -> Result<(), &'static str> { if segment.is_empty() { - return Err(GlobalNameError { - kind: GlobalNameErrorKind::Empty, - reason: "segments must not be empty", - }); + return Err("segments must not be empty"); } for byte in segment.bytes() { if byte < 0x20 || byte == 0x7f { - return Err(GlobalNameError { - kind: GlobalNameErrorKind::Control, - reason: "segments must not contain a control character", - }); + return Err("segments must not contain a control character"); } if !matches!(byte, b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.') { - return Err(GlobalNameError { - kind: GlobalNameErrorKind::Charset, - reason: "segments may contain only lowercase ASCII letters, digits, '-', '_', '.'", - }); + return Err("segments may contain only lowercase ASCII letters, digits, '-', '_', '.'"); } } Ok(()) } -/// A stable classification of a [`GlobalNameError`]. +/// The error returned when an id, or a catalog of ids, is invalid. /// -/// `GlobalNameError::kind` returns it. Match on it to branch on why a name -/// was rejected. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -#[non_exhaustive] -pub enum GlobalNameErrorKind { - /// A segment was empty. - Empty, - /// A segment contained a control byte. - Control, - /// A segment contained a character other than a lowercase ASCII letter, a - /// digit, `-`, `_`, or `.` that is not a control byte. - /// - /// This covers uppercase letters, `@`, and non-ASCII characters. - Charset, -} - -/// The error returned when a [`GlobalName`] fails to parse. +/// Parsing a [`GlobalName`], a [`PackId`], a [`ToolId`], a [`ChainId`] or a +/// [`TaskId`], or building a [`ModelId`], returns it for a malformed id. +/// Building a [`ToolCatalog`] or a [`ModelCatalog`] returns it when two +/// entries share an id. Its message names what was rejected and why, such +/// as `invalid Pack id: segments must not be empty`. +/// +/// [`PackId`]: crate::packs::PackId +/// [`ToolId`]: crate::tools::ToolId +/// [`ChainId`]: crate::ids::ChainId +/// [`TaskId`]: crate::ids::TaskId +/// [`ModelId`]: crate::models::ModelId +/// [`ToolCatalog`]: crate::tools::ToolCatalog +/// [`ModelCatalog`]: crate::models::ModelCatalog #[derive(Debug, Clone, PartialEq, Eq)] -#[non_exhaustive] -pub struct GlobalNameError { - /// A stable classification of why the name was rejected. - kind: GlobalNameErrorKind, - /// A human-readable reason. - reason: &'static str, +pub struct IdError { + /// The rendered message. + message: String, } -impl GlobalNameError { - /// Returns the stable classification of this error. - #[must_use] - pub(crate) fn kind(&self) -> GlobalNameErrorKind { - self.kind +impl IdError { + /// Builds an error that renders as `message`. + pub(crate) fn new(message: String) -> IdError { + IdError { message } } } -impl fmt::Display for GlobalNameError { +impl fmt::Display for IdError { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(f, "invalid global name: {}", self.reason) + f.write_str(&self.message) } } -impl std::error::Error for GlobalNameError {} +impl std::error::Error for IdError {} diff --git a/crates/runvil-internal/types/src/packs-tests.rs b/crates/runvil-internal/types/src/packs-tests.rs index 16a83855c..729e3fb66 100644 --- a/crates/runvil-internal/types/src/packs-tests.rs +++ b/crates/runvil-internal/types/src/packs-tests.rs @@ -1,39 +1,56 @@ //! Tests for the Pack identity vocabulary. -use super::{PackId, PackIdErrorKind, RESERVED_PACK_IDS}; +use super::{PackId, RESERVED_PACK_IDS}; use crate::test_support::assert_wire; use crate::tools::ToolId; +const SEGMENT_COUNT: &str = "invalid Pack id: a Pack id is one segment, such as `web`, with no '/'"; +const EMPTY: &str = "invalid Pack id: segments must not be empty"; +const CONTROL: &str = "invalid Pack id: segments must not contain a control character"; +const CHARSET: &str = + "invalid Pack id: segments may contain only lowercase ASCII letters, digits, '-', '_', '.'"; + +fn message_of(input: &str, why: &str) -> String { + PackId::parse(input).expect_err(why).to_string() +} + #[test] fn pack_id_requires_exactly_one_segment() { let id = PackId::parse("web").expect("one segment parses"); assert_eq!(id.to_string(), "web"); - let two = PackId::parse("runvil/web") - .expect_err("a vendor/name pair is a fully qualified name, not a Pack id"); - assert_eq!(two.kind(), PackIdErrorKind::SegmentCount); - - let tool = PackId::parse("web/fetch").expect_err("two segments are a tool id"); - assert_eq!(tool.kind(), PackIdErrorKind::SegmentCount); + assert_eq!( + message_of( + "runvil/web", + "a vendor/name pair is a fully qualified name, not a Pack id" + ), + SEGMENT_COUNT + ); + assert_eq!( + message_of("web/fetch", "two segments are a tool id"), + SEGMENT_COUNT + ); } #[test] fn pack_id_rejects_empty_and_control_segments() { - let empty = PackId::parse("").expect_err("an empty id is rejected"); - assert_eq!(empty.kind(), PackIdErrorKind::Empty); - - let upper = PackId::parse("Web") - .expect_err("comparison is case-sensitive: uppercase is outside the charset"); - assert_eq!(upper.kind(), PackIdErrorKind::Charset); - - let versioned = PackId::parse("web@2").expect_err("v1 is unversioned: '@' is a parse error"); - assert_eq!(versioned.kind(), PackIdErrorKind::Charset); - - let accented = PackId::parse("w\u{e9}b").expect_err("non-ASCII is outside the charset"); - assert_eq!(accented.kind(), PackIdErrorKind::Charset); - - let control = PackId::parse("we\tb").expect_err("a control byte is rejected"); - assert_eq!(control.kind(), PackIdErrorKind::Control); + assert_eq!(message_of("", "an empty id is rejected"), EMPTY); + assert_eq!( + message_of( + "Web", + "comparison is case-sensitive: uppercase is outside the charset" + ), + CHARSET + ); + assert_eq!( + message_of("web@2", "v1 is unversioned: '@' is a parse error"), + CHARSET + ); + assert_eq!( + message_of("w\u{e9}b", "non-ASCII is outside the charset"), + CHARSET + ); + assert_eq!(message_of("we\tb", "a control byte is rejected"), CONTROL); } #[test] @@ -43,7 +60,11 @@ fn pack_id_from_str_parses_like_parse() { let error = "web/fetch" .parse::() .expect_err("two segments are a tool id"); - assert_eq!(error.kind(), PackIdErrorKind::SegmentCount); + assert_eq!( + error, + PackId::parse("web/fetch").expect_err("two segments are a tool id") + ); + assert_eq!(error.to_string(), SEGMENT_COUNT); } #[test] diff --git a/crates/runvil-internal/types/src/packs.rs b/crates/runvil-internal/types/src/packs.rs index d5e09d553..965b9eb48 100644 --- a/crates/runvil-internal/types/src/packs.rs +++ b/crates/runvil-internal/types/src/packs.rs @@ -1,4 +1,4 @@ -//! The Pack identity vocabulary: [`PackId`] and its parse error, the +//! The Pack identity vocabulary: [`PackId`], the //! [`RESERVED_PACK_IDS`] no Pack may take, [`PackRef`], one entry of a //! prompt's `uses:` list, and [`Prelude`], the Lua source a declared Pack //! hands the Engine. @@ -16,7 +16,7 @@ use std::str::FromStr; -use crate::names::{GlobalName, GlobalNameErrorKind}; +use crate::names::{GlobalName, IdError}; use crate::tools::ToolId; #[cfg(test)] @@ -45,20 +45,20 @@ impl PackId { /// Parses a Pack identity, requiring exactly one segment. /// /// # Errors - /// Returns [`PackIdError`] when the id has more than one segment - /// ([`PackIdErrorKind::SegmentCount`]), is empty or has an empty - /// segment ([`PackIdErrorKind::Empty`]), a segment contains a control - /// byte ([`PackIdErrorKind::Control`]), or a segment contains any - /// other character than a lowercase ASCII letter, a digit, `-`, `_`, - /// or `.` ([`PackIdErrorKind::Charset`]). - pub fn parse(name: &str) -> Result { - let name = - GlobalName::parse(name).map_err(|e| PackIdError::from_global_name_kind(e.kind()))?; + /// Returns [`IdError`] when the id has more than one segment, is empty + /// or has an empty segment, a segment contains a control byte, or a + /// segment contains any other character than a lowercase ASCII letter, + /// a digit, `-`, `_`, or `.`. + pub fn parse(name: &str) -> Result { + PackId::validate(name).map_err(|reason| IdError::new(format!("invalid Pack id: {reason}"))) + } + + /// Parses `name` as [`PackId::parse`] does, but returns only the reason + /// it is invalid. + fn validate(name: &str) -> Result { + let name = GlobalName::validate(name)?; if name.segments().len() != 1 { - return Err(PackIdError { - kind: PackIdErrorKind::SegmentCount, - reason: "a Pack id is one segment, such as `web`, with no '/'", - }); + return Err("a Pack id is one segment, such as `web`, with no '/'"); } Ok(PackId(name)) } @@ -92,9 +92,9 @@ impl PackId { } impl FromStr for PackId { - type Err = PackIdError; + type Err = IdError; - fn from_str(name: &str) -> Result { + fn from_str(name: &str) -> Result { PackId::parse(name) } } @@ -136,8 +136,8 @@ impl serde::de::Visitor<'_> for PackIdVisitor { } fn visit_str(self, text: &str) -> Result { - PackId::parse(text) - .map_err(|error| E::custom(format!("invalid Pack id `{text}`: {}", error.reason))) + PackId::validate(text) + .map_err(|reason| E::custom(format!("invalid Pack id `{text}`: {reason}"))) } } @@ -177,8 +177,7 @@ impl PackRef { /// Parses one entry: one or two valid segments. fn parse(text: &str) -> Result { - let name = GlobalName::parse(text) - .map_err(|error| PackIdError::from_global_name_kind(error.kind()).reason)?; + let name = GlobalName::validate(text)?; let (vendor, id) = match name.segments() { [id] => (RUNVIL_VENDOR, id), [vendor, id] => (vendor.as_str(), id), @@ -221,64 +220,6 @@ impl serde::de::Visitor<'_> for PackRefVisitor { } } -/// Classifies a [`PackIdError`] so callers can handle each kind of failure. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -#[non_exhaustive] -pub enum PackIdErrorKind { - /// The id had more than one segment. - SegmentCount, - /// A segment was empty. - Empty, - /// A segment contained a control byte. - Control, - /// A segment contained a character other than a lowercase ASCII letter, - /// a digit, `-`, `_`, or `.` that is not a control byte, such as an - /// uppercase letter, `@`, or a non-ASCII character. - Charset, -} - -/// The reason a [`PackId`] failed to parse. -#[derive(Debug, Clone, PartialEq, Eq)] -#[non_exhaustive] -pub struct PackIdError { - /// A stable classification of why the id was rejected. - kind: PackIdErrorKind, - /// A human-readable reason. - reason: &'static str, -} - -impl PackIdError { - /// Returns the stable classification of this error. - #[must_use] - pub fn kind(&self) -> PackIdErrorKind { - self.kind - } - - /// Maps a global-name rejection onto the Pack-id error vocabulary. - fn from_global_name_kind(global_kind: GlobalNameErrorKind) -> PackIdError { - let (kind, reason) = match global_kind { - GlobalNameErrorKind::Empty => (PackIdErrorKind::Empty, "segments must not be empty"), - GlobalNameErrorKind::Control => ( - PackIdErrorKind::Control, - "segments must not contain a control character", - ), - GlobalNameErrorKind::Charset => ( - PackIdErrorKind::Charset, - "segments may contain only lowercase ASCII letters, digits, '-', '_', '.'", - ), - }; - PackIdError { kind, reason } - } -} - -impl std::fmt::Display for PackIdError { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(f, "invalid Pack id: {}", self.reason) - } -} - -impl std::error::Error for PackIdError {} - /// Holds the Lua source a declared Pack adds to every section's VM. /// /// A prelude defines tables and functions, such as `sh.run(script)`, that diff --git a/crates/runvil-internal/types/src/replay-tests.rs b/crates/runvil-internal/types/src/replay-tests.rs deleted file mode 100644 index d9a5f407c..000000000 --- a/crates/runvil-internal/types/src/replay-tests.rs +++ /dev/null @@ -1,48 +0,0 @@ -//! Tests for replay `Flags` and `ReplayError` rendering. - -use super::{Flags, ReplayError}; -use crate::test_support::assert_wire; - -#[test] -fn flags_start_empty_and_round_trip_their_bits() { - assert!(Flags::EMPTY.is_empty()); - assert_eq!(Flags::default(), Flags::EMPTY); - assert_eq!(Flags::EMPTY.bits(), 0); - - // A recorded run may set a bit this build does not name yet; the set - // preserves it rather than dropping it, so replay can still see it. - let recorded = Flags::from_bits(0b101); - assert!(!recorded.is_empty()); - assert_eq!(recorded.bits(), 0b101); - assert!(recorded.contains(Flags::from_bits(0b001))); - assert!(recorded.contains(Flags::from_bits(0b100))); - assert!(!recorded.contains(Flags::from_bits(0b010))); - assert_eq!(Flags::from_bits(0b001) | Flags::from_bits(0b100), recorded); -} - -#[test] -fn flags_serialize_as_one_integer() { - assert_wire(&Flags::from_bits(6), "6"); - assert_wire(&Flags::EMPTY, "0"); -} - -#[test] -fn replay_errors_name_their_kind_and_detail() { - fn assert_error() {} - assert_error::(); - - let diverged = ReplayError::Nondeterminism { - detail: "task 0.1 issued a chat effect where the record holds a tool call".to_owned(), - }; - assert_eq!( - diverged.to_string(), - "replay diverged from its record: task 0.1 issued a chat effect where the record holds a tool call" - ); - let malformed = ReplayError::Fatal { - detail: "effect 7 has two answers".to_owned(), - }; - assert_eq!( - malformed.to_string(), - "replay record is malformed: effect 7 has two answers" - ); -} diff --git a/crates/runvil-internal/types/src/replay.rs b/crates/runvil-internal/types/src/replay.rs deleted file mode 100644 index e056a6ba5..000000000 --- a/crates/runvil-internal/types/src/replay.rs +++ /dev/null @@ -1,130 +0,0 @@ -//! Holds the replay vocabulary: recorded behavior flags and replay failures. -//! -//! A run is meant to be reproducible from its log: the same run inputs -//! (seed, `started_at`, flags) and the same answers replayed in order -//! produce the same effects and events, each keyed by its -//! [`Provenance`](crate::ids::Provenance). -//! -//! [`Flags`] is how an Engine change that alters a recorded run's -//! behavior stays replayable: it runs the new behavior live and sets its -//! flag, and a later replay honors the flag only if the original run -//! recorded it. The crate-private `ReplayError` keeps "the code under -//! replay diverged" apart from "the record is broken", because each -//! demands a different remedy. - -use std::fmt; -use std::ops::{BitOr, BitOrAssign}; - -use serde::{Deserialize, Serialize}; - -#[cfg(test)] -#[path = "replay-tests.rs"] -mod tests; - -/// Holds the behavior flags recorded with a run. -/// -/// The set is a bitset stored as a single `u32`, both when serialized and -/// in the run record. -/// -/// Each flag is an associated constant `Flags(1 << n)`. Its bit `n` is -/// assigned once and belongs to that flag permanently, even after the -/// behavior the flag gates becomes the only behavior. -/// [`from_bits`](Self::from_bits) and [`bits`](Self::bits) preserve all 32 -/// bits. A record written by a newer Engine keeps every flag when an older -/// reader loads it, including flags added after that reader was built. -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] -#[repr(transparent)] -#[serde(transparent)] -pub struct Flags(u32); - -impl Flags { - /// The empty set, with every bit clear. - pub const EMPTY: Flags = Flags(0); - - /// Returns the set whose bits are exactly `bits`. - /// - /// It keeps the bits of flags added in a newer build. - #[must_use] - pub const fn from_bits(bits: u32) -> Self { - Self(bits) - } - - /// Returns the set as its `u32` bits. - #[must_use] - pub const fn bits(self) -> u32 { - self.0 - } - - /// True when every bit is clear. - #[must_use] - pub const fn is_empty(self) -> bool { - self.0 == 0 - } - - /// True when every flag in `other` is set in `self`. - #[must_use] - pub const fn contains(self, other: Flags) -> bool { - self.0 & other.0 == other.0 - } -} - -impl BitOr for Flags { - type Output = Flags; - - fn bitor(self, rhs: Flags) -> Flags { - Flags(self.0 | rhs.0) - } -} - -impl BitOrAssign for Flags { - fn bitor_assign(&mut self, rhs: Flags) { - self.0 |= rhs.0; - } -} - -/// Classifies why a replay failed. -/// -/// The two kinds are properties of different things. `Nondeterminism` is a -/// property of the code under replay: re-executed against its record, a run -/// or a task issued an effect or event that disagrees with what the record -/// holds at that [`Provenance`](crate::ids::Provenance), so the Engine (or -/// the prompt) is not deterministic where it must be. `Fatal` is a property -/// of the record: the log is malformed or internally inconsistent (an -/// effect with two answers, a sequence gap, an unparseable payload), so -/// there is nothing sound to replay against. -#[derive(Debug, Clone, PartialEq, Eq)] -#[non_exhaustive] -#[cfg_attr( - not(test), - expect( - dead_code, - reason = "no replay reader exists yet; the vocabulary is fixed ahead of it" - ) -)] -pub(crate) enum ReplayError { - /// The re-executed run or task disagreed with its record. - #[non_exhaustive] - Nondeterminism { - /// Where and how the re-execution disagreed. - detail: String, - }, - /// The record itself is malformed or internally inconsistent. - #[non_exhaustive] - Fatal { - /// What is wrong with the record. - detail: String, - }, -} - -impl fmt::Display for ReplayError { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - ReplayError::Nondeterminism { detail } => { - write!(f, "replay diverged from its record: {detail}") - } - ReplayError::Fatal { detail } => write!(f, "replay record is malformed: {detail}"), - } - } -} - -impl std::error::Error for ReplayError {} diff --git a/crates/runvil-internal/types/src/seed-tests.rs b/crates/runvil-internal/types/src/seed-tests.rs index ff30f41f0..487856369 100644 --- a/crates/runvil-internal/types/src/seed-tests.rs +++ b/crates/runvil-internal/types/src/seed-tests.rs @@ -1,6 +1,6 @@ //! Tests for the run seed and the values derived from it: the derivation's -//! known answer, the nonce and run id it yields, the seed's serde form, and -//! its redacted `Debug`. +//! known answer, the nonce it yields, the seed's serde form, and its +//! redacted `Debug`. use std::fmt::Write as _; @@ -18,32 +18,26 @@ fn hex(bytes: &[u8]) -> String { #[test] fn derive_matches_its_known_answer() { - // SHA-256 over `runvil.run-uid.v1` then the bytes 0x00..=0x1f, + // SHA-256 over `runvil.guard-nonce.v1` then the bytes 0x00..=0x1f, // computed once outside this crate. let seed = RunSeed::from_bytes(std::array::from_fn(|i| { u8::try_from(i).expect("an index below 32 fits a byte") })); - let expected = "9fdae9c2e5fc9ce35263235fc1f3c2fdb9ae59aa64b42725d3dbcb12d5fce826"; - assert_eq!(hex(&derive(b"runvil.run-uid.v1", &seed)), expected); + let expected = "0d89f97db656f4554ed91f0dc4bc9a829d2555c04535cceffdda52e6d44b66cf"; + assert_eq!(hex(&derive(b"runvil.guard-nonce.v1", &seed)), expected); assert_eq!( - RunUid::from_seed(&seed).to_string(), - expected, - "the run id is derive under its own tag" + GuardNonce::from_seed(&seed).to_string(), + expected[..32], + "the nonce is the first half of derive under its own tag" ); } #[test] -fn the_nonce_and_run_id_differ_from_each_other_and_the_seed_and_repeat() { +fn the_nonce_differs_from_the_seed_and_repeats() { let mut rng = Seeded::new(SEED); let seed = rng.run_seed(); let nonce = GuardNonce::from_seed(&seed).to_string(); - let uid = RunUid::from_seed(&seed).to_string(); let bytes = hex(seed.as_bytes()); - rng.check( - !uid.starts_with(&nonce), - format_args!("the nonce {nonce} is the run id's prefix {uid}"), - ); - rng.check(uid != bytes, "the run id is the seed's bytes"); rng.check( !bytes.starts_with(&nonce), format_args!("the nonce {nonce} is the seed's prefix"), @@ -53,11 +47,6 @@ fn the_nonce_and_run_id_differ_from_each_other_and_the_seed_and_repeat() { &nonce, "one seed, one nonce", ); - rng.check_eq( - &RunUid::from_seed(&seed), - &RunUid::from_seed(&seed), - "one seed, one run id", - ); } #[test] diff --git a/crates/runvil-internal/types/src/seed.rs b/crates/runvil-internal/types/src/seed.rs index 4e39b58ec..4472e87e2 100644 --- a/crates/runvil-internal/types/src/seed.rs +++ b/crates/runvil-internal/types/src/seed.rs @@ -3,20 +3,16 @@ //! The caller draws a run's [`RunSeed`] from a CSPRNG for a live run and //! passes the recorded one back for a replay. Every value derived from the //! seed goes through `derive()` under its own `runvil..v1` -//! tag: the guard nonce -//! ([`GuardNonce::from_seed`](crate::untrusted::GuardNonce::from_seed)) and -//! the run id ([`RunUid`]). Each derivation is one-way, so a derived value -//! can be shown without revealing the seed or any other value derived from -//! it. +//! tag, as the guard nonce +//! ([`GuardNonce::from_seed`](crate::untrusted::GuardNonce::from_seed)) +//! does. Each derivation is one-way, so a derived value can be shown +//! without revealing the seed or any other value derived from it. use std::fmt; use serde::{Deserialize, Deserializer, Serialize, Serializer}; use sha2::{Digest as _, Sha256}; -/// The domain tag the run id is derived under. -const RUN_UID_DOMAIN: &[u8] = b"runvil.run-uid.v1"; - /// What a run seed's text form must be, for a refusal to name. const REQUIRED: &str = "a run seed is 64 lowercase hex characters"; @@ -87,37 +83,6 @@ pub(crate) fn derive(domain: &[u8], seed: &RunSeed) -> [u8; 32] { out } -/// Identifies a run by 32 bytes derived from its seed. -/// -/// The derivation is tagged `runvil.run-uid.v1`. The id is never the -/// seed itself, so it can appear in logs, records, and views without -/// revealing the seed or the nonce. A replay passes back the recorded seed, -/// so it reproduces the id. `Display` renders 64 lowercase hex characters. -#[derive(Clone, Copy, PartialEq, Eq, Hash)] -pub struct RunUid([u8; 32]); - -impl RunUid { - /// The id of the run whose seed is `seed`. - #[must_use] - pub fn from_seed(seed: &RunSeed) -> RunUid { - RunUid(derive(RUN_UID_DOMAIN, seed)) - } -} - -impl fmt::Display for RunUid { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write_hex(f, &self.0) - } -} - -impl fmt::Debug for RunUid { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - f.write_str("RunUid(")?; - write_hex(f, &self.0)?; - f.write_str(")") - } -} - /// Writes `bytes` as lowercase hex. fn write_hex(f: &mut fmt::Formatter<'_>, bytes: &[u8; 32]) -> fmt::Result { bytes.iter().try_for_each(|byte| write!(f, "{byte:02x}")) diff --git a/crates/runvil-internal/types/src/timestamp.rs b/crates/runvil-internal/types/src/timestamp.rs index f132f5b1e..2efd5799e 100644 --- a/crates/runvil-internal/types/src/timestamp.rs +++ b/crates/runvil-internal/types/src/timestamp.rs @@ -1,6 +1,6 @@ //! A UTC instant in milliseconds, rendered to RFC 3339 over std alone. //! -//! A run's `started_at` is an input the caller passes to `RunContext::new`; +//! A run's `started_at` is an input the caller passes to `RunConfig::new`; //! the clock belongs to the Harness. //! [`Timestamp`] is the value that crosses that boundary. Its one //! rendering, [`to_rfc3339`](Timestamp::to_rfc3339), is what a prompt diff --git a/crates/runvil-internal/types/src/tools.rs b/crates/runvil-internal/types/src/tools.rs index e0df84e1b..a47069aae 100644 --- a/crates/runvil-internal/types/src/tools.rs +++ b/crates/runvil-internal/types/src/tools.rs @@ -8,11 +8,11 @@ //! This module holds vocabulary only: the implementation-free //! [`ToolInfo`] and the caller-supplied [`ToolCatalog`] of `ToolInfo` values, //! trusted output ([`ToolOutput`], [`OutputTrust`]), the model-safe -//! [`ToolError`], a call's [`ToolCallOrigin`], and the contract errors. The -//! implementation trait behind a tool info (`Pack`) is the Plugin -//! contract's, in `runvil-plugin`, which the `plugin-*` crates -//! implement; the prompt parser and -//! the executor sit in their own crates and depend on `runvil-types`. +//! [`ToolError`], and a call's [`ToolCallOrigin`]. The implementation +//! trait behind a tool info (`Pack`) is the Plugin contract's, in +//! `runvil-plugin`, which the `plugin-*` crates implement; the prompt +//! parser and the executor sit in their own crates and depend on +//! `runvil-types`. mod ids; mod info; @@ -20,11 +20,11 @@ mod origin; mod output; mod registry; -pub use ids::{ToolId, ToolIdError, ToolIdErrorKind}; +pub use ids::ToolId; pub use info::ToolInfo; pub use origin::{ToolCallOrigin, ToolCaller}; pub use output::{OutputTrust, ToolError, ToolErrorKind, ToolOutput}; -pub use registry::{ToolCatalog, ToolCatalogError, ToolCatalogErrorKind}; +pub use registry::ToolCatalog; #[cfg(test)] mod tests; diff --git a/crates/runvil-internal/types/src/tools/ids.rs b/crates/runvil-internal/types/src/tools/ids.rs index 9aa7e2ab8..6b6ac12aa 100644 --- a/crates/runvil-internal/types/src/tools/ids.rs +++ b/crates/runvil-internal/types/src/tools/ids.rs @@ -1,8 +1,8 @@ -//! Stable tool identity and its validation errors. +//! Stable tool identity. use std::str::FromStr; -use crate::names::{GlobalName, GlobalNameErrorKind}; +use crate::names::{GlobalName, IdError}; use crate::packs::PackId; /// Identifies a tool by its Pack's local name and the Pack's own segments. @@ -24,21 +24,17 @@ impl ToolId { /// Parses a tool id of two or more segments (`pack/name`). /// /// # Errors - /// Returns [`ToolIdError`] when the id has fewer than two segments - /// ([`ToolIdErrorKind::SegmentCount`]), is empty or has an empty segment - /// ([`ToolIdErrorKind::Empty`]), a segment contains a control byte - /// ([`ToolIdErrorKind::Control`]), or a segment contains any other - /// character outside the set of lowercase ASCII letters, digits, `-`, - /// `_`, and `.` ([`ToolIdErrorKind::Charset`]). - pub fn parse(id: &str) -> Result { - let name = - GlobalName::parse(id).map_err(|e| ToolIdError::from_global_name_kind(e.kind()))?; + /// Returns [`IdError`] when the id has fewer than two segments, is + /// empty or has an empty segment, a segment contains a control byte, or + /// a segment contains any other character outside the set of lowercase + /// ASCII letters, digits, `-`, `_`, and `.`. + pub fn parse(id: &str) -> Result { + let invalid = |reason: &str| IdError::new(format!("invalid tool id: {reason}")); + let name = GlobalName::validate(id).map_err(invalid)?; if name.segments().len() < 2 { - return Err(ToolIdError { - field: "id", - kind: ToolIdErrorKind::SegmentCount, - reason: "a tool id has two or more segments, its Pack's name first, such as `web/fetch`", - }); + return Err(invalid( + "a tool id has two or more segments, its Pack's name first, such as `web/fetch`", + )); } Ok(ToolId(name)) } @@ -71,9 +67,9 @@ impl ToolId { } impl FromStr for ToolId { - type Err = ToolIdError; + type Err = IdError; - fn from_str(id: &str) -> Result { + fn from_str(id: &str) -> Result { ToolId::parse(id) } } @@ -103,70 +99,3 @@ impl<'de> serde::Deserialize<'de> for ToolId { ToolId::parse(&text).map_err(serde::de::Error::custom) } } - -/// The stable category of a [`ToolIdError`], which callers can match on. -/// -/// Get it from `ToolIdError::kind` to handle each kind of rejection -/// differently. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -#[non_exhaustive] -pub enum ToolIdErrorKind { - /// The id had fewer than two segments. - SegmentCount, - /// The id was empty or had an empty segment. - Empty, - /// A segment contained a control byte. - Control, - /// A segment contained a character outside the allowed set that is not - /// a control byte, such as an uppercase letter, `@`, or a non-ASCII - /// character. - Charset, -} - -/// The reason a [`ToolId`] failed to parse. -#[derive(Debug, Clone, PartialEq, Eq)] -#[non_exhaustive] -pub struct ToolIdError { - /// What was rejected: `id`, the tool id being parsed. - field: &'static str, - /// A stable classification of why it was rejected. - kind: ToolIdErrorKind, - /// A human-readable reason. - reason: &'static str, -} - -impl ToolIdError { - /// Returns the stable classification of this error. - #[must_use] - pub fn kind(&self) -> ToolIdErrorKind { - self.kind - } - - /// Maps a global-name rejection onto the tool-id error vocabulary. - fn from_global_name_kind(global_kind: GlobalNameErrorKind) -> ToolIdError { - let (kind, reason) = match global_kind { - GlobalNameErrorKind::Empty => (ToolIdErrorKind::Empty, "segments must not be empty"), - GlobalNameErrorKind::Control => ( - ToolIdErrorKind::Control, - "segments must not contain a control character", - ), - GlobalNameErrorKind::Charset => ( - ToolIdErrorKind::Charset, - "segments may contain only lowercase ASCII letters, digits, '-', '_', '.'", - ), - }; - ToolIdError { - field: "id", - kind, - reason, - } - } -} - -impl std::fmt::Display for ToolIdError { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(f, "invalid tool {}: {}", self.field, self.reason) - } -} - -impl std::error::Error for ToolIdError {} diff --git a/crates/runvil-internal/types/src/tools/info.rs b/crates/runvil-internal/types/src/tools/info.rs index cbcf77406..54b4f5559 100644 --- a/crates/runvil-internal/types/src/tools/info.rs +++ b/crates/runvil-internal/types/src/tools/info.rs @@ -11,12 +11,14 @@ use super::ids::ToolId; /// A `ToolInfo` holds the tool's stable id, the name the model calls it by /// when one is set, the description the model reads, its parameter schema, /// whether its output is structured, whether a stop leaves its calls -/// running, and whether its output lists tools. +/// running, whether its output lists tools, and whether only a model may +/// call it. /// /// The caller collects the `ToolInfo` values of every Pack it can serve /// into a [`ToolCatalog`](super::ToolCatalog). The implementations stay with /// the caller, keyed by [`ToolId`]. Every catalog tool is available to the -/// prompt's Lua, which calls one by id, and a section offers one to its +/// prompt's Lua, which calls one by id unless the tool is model-only, and a +/// section offers one to its /// model with `tools.offer`. The Engine issues each tool call as an effect /// that names the tool's id. The caller resolves that id to the /// implementation. @@ -30,6 +32,10 @@ use super::ids::ToolId; /// cannot see or call. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[non_exhaustive] +#[expect( + clippy::struct_excessive_bools, + reason = "the four flags are independent: each answers a different question about one tool, so no state machine or enum expresses them" +)] pub struct ToolInfo { /// The tool's stable identity. The catalog uses it as the key. pub id: ToolId, @@ -62,6 +68,18 @@ pub struct ToolInfo { /// `false`. #[serde(default)] pub describes_tools: bool, + /// Whether only a model may call the tool, so a script's `tools.call` + /// refuses it. + /// + /// A Pack sets it when a render gives the tool arguments a script + /// cannot know: a script names a tool by id and passes JSON it wrote, + /// while the arguments a model sends follow whatever schema the + /// section's render answered for that model. The tool stays in the + /// catalog, so Lua names it and a section offers it with + /// `tools.offer`; only the call is refused. A `ToolInfo` serialized + /// without the field reads as `false`. + #[serde(default)] + pub model_only_call: bool, } impl ToolInfo { @@ -77,6 +95,7 @@ impl ToolInfo { structured_output: false, survives_stop: false, describes_tools: false, + model_only_call: false, } } @@ -110,4 +129,12 @@ impl ToolInfo { self.describes_tools = describes; self } + + /// Sets whether only a model may call the tool (`true`), so a script's + /// `tools.call` refuses it, or a script may call it too (`false`). + #[must_use] + pub fn model_only_call(mut self, model_only: bool) -> ToolInfo { + self.model_only_call = model_only; + self + } } diff --git a/crates/runvil-internal/types/src/tools/output.rs b/crates/runvil-internal/types/src/tools/output.rs index 100f99512..8ca8d574a 100644 --- a/crates/runvil-internal/types/src/tools/output.rs +++ b/crates/runvil-internal/types/src/tools/output.rs @@ -70,8 +70,6 @@ pub enum ToolErrorKind { /// The request failed at the transport layer, such as a network error or a /// timeout. Transport, - /// The run was cancelled before or during the call. - Cancelled, /// Any other tool failure. Other, } @@ -85,7 +83,6 @@ impl ToolErrorKind { ToolErrorKind::InvalidArguments => "invalid_arguments", ToolErrorKind::Backend => "backend", ToolErrorKind::Transport => "transport", - ToolErrorKind::Cancelled => "cancelled", ToolErrorKind::Other => "other", } } @@ -147,18 +144,6 @@ impl ToolError { pub fn kind(&self) -> ToolErrorKind { self.kind } - - /// Returns whether the failure was a cancellation. - #[must_use] - pub fn is_cancelled(&self) -> bool { - matches!(self.kind, ToolErrorKind::Cancelled) - } - - /// Returns whether retrying the same call could plausibly succeed. - #[must_use] - pub fn is_retryable(&self) -> bool { - matches!(self.kind, ToolErrorKind::Transport) - } } impl std::fmt::Display for ToolError { diff --git a/crates/runvil-internal/types/src/tools/registry.rs b/crates/runvil-internal/types/src/tools/registry.rs index 7beab0a98..e715d61bb 100644 --- a/crates/runvil-internal/types/src/tools/registry.rs +++ b/crates/runvil-internal/types/src/tools/registry.rs @@ -1,11 +1,11 @@ -//! The caller-supplied [`ToolCatalog`] of `ToolInfo` values and the catalog's -//! construction error. +//! The caller-supplied [`ToolCatalog`] of `ToolInfo` values. use std::fmt; use std::sync::Arc; use super::ids::ToolId; use super::info::ToolInfo; +use crate::names::IdError; /// A catalog of the tools a run may offer, given as `ToolInfo` values. /// @@ -45,16 +45,16 @@ impl ToolCatalog { /// This is the only place the catalog checks identities. /// /// # Errors - /// Returns [`ToolCatalogError::DuplicateId`] if two `ToolInfo` values - /// share a [`ToolId`]. - pub fn new(tools: impl IntoIterator) -> Result { + /// Returns [`IdError`] if two `ToolInfo` values share a [`ToolId`]. + pub fn new(tools: impl IntoIterator) -> Result { let tools: Vec = tools.into_iter().collect(); let mut seen = std::collections::BTreeSet::new(); for tool in &tools { if !seen.insert(&tool.id) { - return Err(ToolCatalogError::DuplicateId { - id: tool.id.clone(), - }); + return Err(IdError::new(format!( + "duplicate tool identity `{}` in the tool catalog", + tool.id + ))); } } Ok(Self { @@ -76,56 +76,3 @@ impl ToolCatalog { &self.tools } } - -/// A stable, matchable classification of a [`ToolCatalogError`]. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -#[non_exhaustive] -pub enum ToolCatalogErrorKind { - /// Two supplied tools shared a stable [`ToolId`]. - DuplicateId, -} - -/// Reports why building a [`ToolCatalog`] failed. -/// -/// Building fails when two tools share an identity. Call -/// [`kind`](Self::kind) to get a stable classification you can match on. -#[derive(Debug, Clone, PartialEq, Eq)] -#[non_exhaustive] -pub enum ToolCatalogError { - /// The same stable identity was supplied by more than one tool. - #[non_exhaustive] - DuplicateId { - /// The stable identity supplied more than once. - id: ToolId, - }, -} - -impl ToolCatalogError { - /// Returns the stable classification of this error. - #[must_use] - pub fn kind(&self) -> ToolCatalogErrorKind { - match self { - ToolCatalogError::DuplicateId { .. } => ToolCatalogErrorKind::DuplicateId, - } - } - - /// Returns the duplicated identity when this is a [`Self::DuplicateId`]. - #[must_use] - pub fn duplicate_id(&self) -> Option<&ToolId> { - match self { - ToolCatalogError::DuplicateId { id } => Some(id), - } - } -} - -impl std::fmt::Display for ToolCatalogError { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - ToolCatalogError::DuplicateId { id } => { - write!(f, "duplicate tool identity `{id}` in the tool catalog") - } - } - } -} - -impl std::error::Error for ToolCatalogError {} diff --git a/crates/runvil-internal/types/src/tools/tests.rs b/crates/runvil-internal/types/src/tools/tests.rs index 24d4d8ada..ae1da824d 100644 --- a/crates/runvil-internal/types/src/tools/tests.rs +++ b/crates/runvil-internal/types/src/tools/tests.rs @@ -2,7 +2,7 @@ use serde_json::json; -use super::{ToolCatalog, ToolCatalogErrorKind, ToolId, ToolInfo}; +use super::{ToolCatalog, ToolId, ToolInfo}; use crate::packs::PackId; use crate::test_support::{assert_send_sync, assert_wire, round_trip}; @@ -55,7 +55,6 @@ fn each_tool_error_kind_tags_as_its_snake_case_name() { (ToolErrorKind::InvalidArguments, "invalid_arguments"), (ToolErrorKind::Backend, "backend"), (ToolErrorKind::Transport, "transport"), - (ToolErrorKind::Cancelled, "cancelled"), (ToolErrorKind::Other, "other"), ] { assert_eq!(kind.tag(), tag); @@ -78,13 +77,9 @@ fn tool_error_classifies_and_hides_source() { let plain = ToolError::message("model-safe"); assert_eq!(plain.kind(), ToolErrorKind::Other); assert_eq!(plain.to_string(), "model-safe"); - assert!(!plain.is_cancelled() && !plain.is_retryable()); - let cancelled = ToolError::message("stopped").with_kind(ToolErrorKind::Cancelled); - assert!(cancelled.is_cancelled()); - - let retry = ToolError::message("net").with_kind(ToolErrorKind::Transport); - assert!(retry.is_retryable()); + let transport = ToolError::message("net").with_kind(ToolErrorKind::Transport); + assert_eq!(transport.kind(), ToolErrorKind::Transport); let sourced = ToolError::message("wrap").with_source(std::io::Error::other("cause")); assert!(std::error::Error::source(&sourced).is_some()); @@ -158,15 +153,16 @@ fn survives_stop_sets_the_flag_and_round_trips_through_serde() { } #[test] -fn a_tool_info_logged_before_wire_name_existed_reads_it_as_absent_and_logs_the_same_line() { - let line = r#"{"id":"fixtures/tools/inspect","description":"Inspect a fixture.","parameters_schema":{"properties":{"path":{"type":"string"}},"required":["path"],"type":"object"},"structured_output":false,"survives_stop":false,"describes_tools":false}"#; +fn a_tool_info_logged_before_wire_name_existed_reads_absent_flags_as_false() { + let line = r#"{"id":"fixtures/tools/inspect","description":"Inspect a fixture.","parameters_schema":{"properties":{"path":{"type":"string"}},"required":["path"],"type":"object"},"structured_output":false,"survives_stop":false}"#; let info: ToolInfo = serde_json::from_str(line).expect("a tool info from before the field deserializes"); assert_eq!( info.wire_name, None, "an absent wire name reads as none, so the model sees the default" ); - assert_wire(&info, line); + assert!(!info.describes_tools, "an absent flag reads as false"); + assert!(!info.model_only_call, "an absent flag reads as false"); } #[test] @@ -176,13 +172,26 @@ fn a_wire_name_its_builder_sets_round_trips_exactly_in_canonical_key_order() { let canonical = serde_json::to_value(&info).expect("a tool info converts to a JSON value"); assert_wire( &canonical, - r#"{"describes_tools":false,"description":"Inspect a fixture.","id":"fixtures/tools/inspect","parameters_schema":{"properties":{"path":{"type":"string"}},"required":["path"],"type":"object"},"structured_output":false,"survives_stop":false,"wire_name":"inspect"}"#, + r#"{"describes_tools":false,"description":"Inspect a fixture.","id":"fixtures/tools/inspect","model_only_call":false,"parameters_schema":{"properties":{"path":{"type":"string"}},"required":["path"],"type":"object"},"structured_output":false,"survives_stop":false,"wire_name":"inspect"}"#, ); let back: ToolInfo = serde_json::from_value(canonical).expect("the canonical form reads back"); assert_eq!(back, info); assert_eq!(round_trip(&info), info); } +#[test] +fn model_only_call_sets_the_flag_and_round_trips_through_serde() { + assert!( + !inspect_info().model_only_call, + "a new tool info is callable by a script" + ); + let info = inspect_info().model_only_call(true); + assert!(info.model_only_call); + assert_eq!(round_trip(&info), info); + let canonical = serde_json::to_value(&info).expect("a tool info converts to a JSON value"); + assert_eq!(canonical["model_only_call"], json!(true)); +} + #[test] fn a_tool_call_origin_keeps_its_logged_shape() { use super::{ToolCallOrigin, ToolCaller}; @@ -258,22 +267,23 @@ fn catalog_preserves_order_and_first_match_lookup() { fn catalog_rejects_duplicate_tool_ids() { let error = ToolCatalog::new([catalog_info("inspect"), catalog_info("inspect")]) .expect_err("a repeated tool identity must be rejected at catalog construction"); - assert_eq!(error.kind(), ToolCatalogErrorKind::DuplicateId); - assert_eq!( - error.duplicate_id(), - Some(&inspect_id()), - "the error must name the duplicated identity" - ); assert_eq!( error.to_string(), - "duplicate tool identity `fixtures/tools/inspect` in the tool catalog" + "duplicate tool identity `fixtures/tools/inspect` in the tool catalog", + "the error must name the duplicated identity" ); } -fn tool_id_error_kind(input: &str) -> super::ToolIdErrorKind { +const SEGMENT_COUNT: &str = "invalid tool id: a tool id has two or more segments, its Pack's name first, such as `web/fetch`"; +const EMPTY: &str = "invalid tool id: segments must not be empty"; +const CONTROL: &str = "invalid tool id: segments must not contain a control character"; +const CHARSET: &str = + "invalid tool id: segments may contain only lowercase ASCII letters, digits, '-', '_', '.'"; + +fn tool_id_error(input: &str) -> String { ToolId::parse(input) .expect_err("the input must be rejected") - .kind() + .to_string() } #[test] @@ -308,57 +318,48 @@ fn four_segments_parse_as_a_tool_id() { #[test] fn a_single_segment_is_rejected_as_a_tool_id() { - use super::ToolIdErrorKind; - assert_eq!(tool_id_error_kind("web"), ToolIdErrorKind::SegmentCount); + assert_eq!(tool_id_error("web"), SEGMENT_COUNT); } #[test] fn an_empty_tool_id_is_rejected_as_an_empty_error() { - use super::ToolIdErrorKind; - assert_eq!(tool_id_error_kind(""), ToolIdErrorKind::Empty); + assert_eq!(tool_id_error(""), EMPTY); } #[test] fn an_empty_segment_is_rejected_as_an_empty_error() { - use super::ToolIdErrorKind; - assert_eq!(tool_id_error_kind("web//fetch"), ToolIdErrorKind::Empty); + assert_eq!(tool_id_error("web//fetch"), EMPTY); } #[test] fn a_control_character_is_rejected_as_a_control_error() { - use super::ToolIdErrorKind; - assert_eq!(tool_id_error_kind("we\tb/fetch"), ToolIdErrorKind::Control); - assert_eq!( - tool_id_error_kind("web/fe\u{7f}tch"), - ToolIdErrorKind::Control - ); + assert_eq!(tool_id_error("we\tb/fetch"), CONTROL); + assert_eq!(tool_id_error("web/fe\u{7f}tch"), CONTROL); } #[test] fn an_uppercase_segment_is_rejected_as_a_charset_error_because_comparison_is_case_sensitive() { - use super::ToolIdErrorKind; - assert_eq!(tool_id_error_kind("Web/fetch"), ToolIdErrorKind::Charset); + assert_eq!(tool_id_error("Web/fetch"), CHARSET); } #[test] fn an_at_sign_or_non_ascii_segment_is_rejected_as_a_charset_error() { - use super::ToolIdErrorKind; - assert_eq!(tool_id_error_kind("web@2/fetch"), ToolIdErrorKind::Charset); - assert_eq!( - tool_id_error_kind("web/f\u{e9}tch"), - ToolIdErrorKind::Charset - ); + assert_eq!(tool_id_error("web@2/fetch"), CHARSET); + assert_eq!(tool_id_error("web/f\u{e9}tch"), CHARSET); } #[test] fn a_tool_id_from_str_parses_like_parse() { - use super::ToolIdErrorKind; let parsed: ToolId = "web/fetch".parse().expect("a valid tool id parses"); assert_eq!(parsed, ToolId::parse("web/fetch").expect("a valid tool id")); let error = "web" .parse::() .expect_err("one segment is rejected"); - assert_eq!(error.kind(), ToolIdErrorKind::SegmentCount); + assert_eq!( + error, + ToolId::parse("web").expect_err("one segment is rejected") + ); + assert_eq!(error.to_string(), SEGMENT_COUNT); } #[test] diff --git a/crates/runvil-internal/vfs/src/handle/operations.rs b/crates/runvil-internal/vfs/src/handle/operations.rs index 7b754fd70..c1691f241 100644 --- a/crates/runvil-internal/vfs/src/handle/operations.rs +++ b/crates/runvil-internal/vfs/src/handle/operations.rs @@ -36,6 +36,26 @@ impl Access { }) } + /// Reads up to `len` bytes of the file at `path`, starting at byte + /// `offset`. + /// + /// It returns fewer than `len` bytes when the file ends first, and + /// nothing when `offset` is at or past the end. `RealBackend` seeks + /// and reads only the requested bytes. A backend without an override + /// reads the whole file and slices it, so the read is not bounded + /// there. + /// + /// # Errors + /// Returns an error when the policy denies the read, when another + /// access holds a conflicting claim on `path` that is not ordered + /// before this operation, or when the backend fails. + pub fn read_range(&self, path: &str, offset: u64, len: u64) -> Result, VfsError> { + let path = self.gate(Op::Read, path)?; + self.admit(Claims::claim_read, &path)?; + self.fire(Op::Read, &path); + self.inner().read_range(&path, offset, len) + } + /// Creates or overwrites the file at `path` with `contents`. /// /// # Errors diff --git a/crates/runvil-internal/vfs/src/handle/tests.rs b/crates/runvil-internal/vfs/src/handle/tests.rs index 33b4372a8..042ee8db9 100644 --- a/crates/runvil-internal/vfs/src/handle/tests.rs +++ b/crates/runvil-internal/vfs/src/handle/tests.rs @@ -17,6 +17,7 @@ use crate::traits::{AcquireContext, Op, Policy, Verdict}; mod claims; mod happens_before; mod operations; +mod read_range; mod refusals; mod regions; /// The rooted-path, idempotent-remove, and split-glob semantics of diff --git a/crates/runvil-internal/vfs/src/handle/tests/read_range.rs b/crates/runvil-internal/vfs/src/handle/tests/read_range.rs new file mode 100644 index 000000000..5e120b86b --- /dev/null +++ b/crates/runvil-internal/vfs/src/handle/tests/read_range.rs @@ -0,0 +1,240 @@ +//! Tests for `Access::read_range`: it takes the same four steps as +//! `read`, it passes its range through to the backend untouched, and it +//! slices on the memory backend. + +use super::*; +use crate::MemoryBackend; +use crate::stat::{Entry, Stat}; +use crate::traits::{ExecId, Vfs, VfsAccess}; + +/// A policy that denies reads and allows everything else, with a reason +/// that names the path. +struct DenyReads; + +impl Policy for DenyReads { + fn check(&self, op: Op, path: &VfsPath) -> Verdict { + if op == Op::Read { + Verdict::Deny(format!("reads of {path} are sealed")) + } else { + Verdict::Allow + } + } +} + +/// The `(offset, len)` pairs a [`RangeSpy`] received, in arrival order. +type Ranges = Arc>>; + +/// A backend over memory whose sessions record the range each +/// `read_range` asks for, then answer it from storage. +struct RangeSpy { + memory: MemoryBackend, + ranges: Ranges, +} + +impl Vfs for RangeSpy { + fn acquire(&mut self, cx: &AcquireContext) -> Result, VfsError> { + Ok(Box::new(RangeSpyAccess { + storage: self.memory.acquire(cx)?, + ranges: Arc::clone(&self.ranges), + })) + } + + fn release(&mut self, id: ExecId) -> Result<(), VfsError> { + self.memory.release(id) + } +} + +/// One session with a [`RangeSpy`]: the required methods forward to +/// storage, and `read_range` records its range first. +struct RangeSpyAccess { + storage: Box, + ranges: Ranges, +} + +impl VfsAccess for RangeSpyAccess { + fn read(&self, path: &VfsPath) -> Result, VfsError> { + self.storage.read(path) + } + + fn read_range(&self, path: &VfsPath, offset: u64, len: u64) -> Result, VfsError> { + self.ranges + .lock() + .unwrap_or_else(PoisonError::into_inner) + .push((offset, len)); + self.storage.read_range(path, offset, len) + } + + fn write(&mut self, path: &VfsPath, contents: &[u8]) -> Result<(), VfsError> { + self.storage.write(path, contents) + } + + fn append(&mut self, path: &VfsPath, contents: &[u8]) -> Result<(), VfsError> { + self.storage.append(path, contents) + } + + fn remove(&mut self, path: &VfsPath, recursive: bool) -> Result<(), VfsError> { + self.storage.remove(path, recursive) + } + + fn exists(&self, path: &VfsPath) -> Result { + self.storage.exists(path) + } + + fn glob(&self, pattern: &str) -> Result, VfsError> { + self.storage.glob(pattern) + } + + fn list(&self, path: &VfsPath) -> Result, VfsError> { + self.storage.list(path) + } + + fn stat(&self, path: &VfsPath) -> Result { + self.storage.stat(path) + } + + fn mkdir(&mut self, path: &VfsPath, recursive: bool) -> Result<(), VfsError> { + self.storage.mkdir(path, recursive) + } + + fn rename(&mut self, from: &VfsPath, to: &VfsPath) -> Result<(), VfsError> { + self.storage.rename(from, to) + } + + fn copy(&mut self, from: &VfsPath, to: &VfsPath) -> Result<(), VfsError> { + self.storage.copy(from, to) + } +} + +#[test] +fn a_policy_denied_read_range_never_reaches_the_backend() -> Result<(), VfsError> { + let fs = TestFs::seeded(&[("/f.txt", "hello world")]); + let vfs = VfsRef::builder() + .mount("/", fs.clone()) + .policy(DenyReads) + .build(); + let access = access(&vfs)?; + assert_eq!( + access.read_range("/f.txt", 0, 10), + Err(VfsError::PermissionDenied { + path: "/f.txt".to_owned(), + reason: "reads of /f.txt are sealed".to_owned(), + }) + ); + let reads: Vec = fs + .calls() + .into_iter() + .filter(|call| matches!(call, Call::Op(Op::Read, _))) + .collect(); + assert!(reads.is_empty(), "a denied read_range reached the backend"); + Ok(()) +} + +#[test] +fn a_read_range_takes_a_read_claim() -> Result<(), VfsError> { + let vfs = handle(TestFs::seeded(&[("/f.txt", "data")])); + let reader = access(&vfs)?; + let writer = access(&vfs)?; + reader.read_range("/f.txt", 0, 4)?; + let message = conflict_message(writer.write("/f.txt", b"new")); + assert!(message.contains("/f.txt"), "names the path: {message}"); + assert!( + message.contains("read"), + "names the standing claim kind: {message}" + ); + Ok(()) +} + +#[test] +fn read_ranges_by_two_identities_never_conflict() -> Result<(), VfsError> { + let vfs = handle(TestFs::seeded(&[("/f.txt", "data")])); + let first = access(&vfs)?; + let second = access(&vfs)?; + first.read_range("/f.txt", 0, 2)?; + assert_eq!(second.read_range("/f.txt", 2, 2)?, b"ta"); + Ok(()) +} + +#[test] +fn a_read_range_fires_one_read_event_and_a_denied_one_fires_none() -> Result<(), VfsError> { + let events: Arc>> = Arc::new(Mutex::new(Vec::new())); + let recorded = Arc::clone(&events); + let vfs = VfsRef::builder() + .mount("/", TestFs::seeded(&[("/a/f.txt", "data")])) + .on_op(move |event: OpEvent<'_>| { + recorded + .lock() + .unwrap_or_else(PoisonError::into_inner) + .push((event.op(), event.path().to_string())); + }) + .build(); + let caller = access(&vfs)?; + // The path is spelled with an alias, so the event carries the + // canonical spelling. + caller.read_range("/a/./f.txt", 0, 4)?; + assert_eq!( + events + .lock() + .unwrap_or_else(PoisonError::into_inner) + .as_slice(), + &[(Op::Read, "/a/f.txt".to_owned())] + ); + + let denied_events: Arc>> = Arc::new(Mutex::new(Vec::new())); + let recorded = Arc::clone(&denied_events); + let sealed = VfsRef::builder() + .mount("/", TestFs::seeded(&[("/f.txt", "data")])) + .policy(DenyReads) + .on_op(move |event: OpEvent<'_>| { + recorded + .lock() + .unwrap_or_else(PoisonError::into_inner) + .push(event.op()); + }) + .build(); + let denied = access(&sealed)?; + assert!(matches!( + denied.read_range("/f.txt", 0, 4), + Err(VfsError::PermissionDenied { .. }) + )); + assert!( + denied_events + .lock() + .unwrap_or_else(PoisonError::into_inner) + .is_empty(), + "a denied read_range fired the sink" + ); + Ok(()) +} + +#[test] +fn a_read_range_passes_its_offset_and_length_through_unclamped() -> Result<(), VfsError> { + let ranges: Ranges = Arc::new(Mutex::new(Vec::new())); + let vfs = handle(RangeSpy { + memory: MemoryBackend::new(), + ranges: Arc::clone(&ranges), + }); + let access = access(&vfs)?; + access.write("/f.txt", b"hello")?; + // A length one past 16 MiB: were it clamped, or turned into a + // whole-file read, the backend would not record it. + let bytes = access.read_range("/f.txt", 3, 16_777_217)?; + assert_eq!(bytes, b"lo"); + assert_eq!( + ranges + .lock() + .unwrap_or_else(PoisonError::into_inner) + .as_slice(), + &[(3, 16_777_217)] + ); + Ok(()) +} + +#[test] +fn a_read_range_slices_on_the_memory_backend() -> Result<(), VfsError> { + let access = access(&VfsRef::default())?; + access.write("/f.txt", b"hello world")?; + assert_eq!(access.read_range("/f.txt", 6, 5)?, b"world"); + assert_eq!(access.read_range("/f.txt", 6, 100)?, b"world"); + assert!(access.read_range("/f.txt", 100, 5)?.is_empty()); + Ok(()) +} diff --git a/crates/runvil-internal/vfs/src/router-access.rs b/crates/runvil-internal/vfs/src/router-access.rs index bf6efff4f..b46f39495 100644 --- a/crates/runvil-internal/vfs/src/router-access.rs +++ b/crates/runvil-internal/vfs/src/router-access.rs @@ -3,13 +3,13 @@ //! backend's session lazily on first touch. use std::cell::RefCell; -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use std::sync::Arc; use super::{Mounts, lock, rejoin, resolve, strip_mount}; use crate::error::VfsError; use crate::path::{VfsPath, VfsPathBuf, canonicalize_absolute}; -use crate::stat::{Entry, Stat}; +use crate::stat::{Entry, FileType, Stat}; use crate::traits::{AcquireContext, VfsAccess}; /// One identity's session with the router. Resolves the longest-prefix @@ -57,6 +57,31 @@ impl RoutingAccess { canonicalize_absolute(strip_mount(prefix.as_str(), path.as_str())) } + /// The names of the directories that `path` has only because a mount + /// sits beneath it: the next path segment of every mount prefix below + /// `path`, sorted and without repeats. A mount at `/a/b` gives `/` the + /// name `a` and `/a` the name `b`, and gives `/a/b` and `/a/b/c` none. + fn mount_children(&self, path: &VfsPath) -> Vec { + let below = path.as_str(); + let mut names = BTreeSet::new(); + for prefix in self.mounts.keys() { + let rest = if below == "/" { + prefix.as_str().strip_prefix('/') + } else { + prefix + .as_str() + .strip_prefix(below) + .and_then(|rest| rest.strip_prefix('/')) + }; + if let Some(name) = rest.and_then(|rest| rest.split('/').next()) + && !name.is_empty() + { + names.insert(name.to_owned()); + } + } + names.into_iter().collect() + } + /// Rejects mutations on read-only mounts before anything is /// touched: a denied operation never partially applies. fn check_writable(&self, path: &VfsPath) -> Result<(), VfsError> { @@ -154,7 +179,10 @@ impl VfsAccess for RoutingAccess { fn exists(&self, path: &VfsPath) -> Result { let stripped = self.strip(path)?; - self.with_mount(path, |session| session.exists(&stripped)) + let served = self.with_mount(path, |session| session.exists(&stripped))?; + // A directory a mount sits under exists, whatever its serving + // backend says. + Ok(served || !self.mount_children(path).is_empty()) } fn glob(&self, pattern: &str) -> Result, VfsError> { @@ -173,12 +201,39 @@ impl VfsAccess for RoutingAccess { fn list(&self, path: &VfsPath) -> Result, VfsError> { let stripped = self.strip(path)?; - self.with_mount(path, |session| session.list(&stripped)) + let served = self.with_mount(path, |session| session.list(&stripped)); + let children = self.mount_children(path); + if children.is_empty() { + return served; + } + // A directory a mount sits under lists the mount's next segment + // as a directory, beside what its serving backend holds, and + // exists as an empty directory when that backend holds nothing. + let mut entries = match served { + Ok(entries) => entries, + Err(VfsError::NotFound { .. }) => Vec::new(), + Err(other) => return Err(other), + }; + let before = entries.len(); + for name in children { + if entries.iter().all(|entry| entry.name != name) { + entries.push(Entry::new(name, Stat::new(FileType::Directory, 0))); + } + } + if entries.len() != before { + entries.sort_by(|left, right| left.name.cmp(&right.name)); + } + Ok(entries) } fn stat(&self, path: &VfsPath) -> Result { let stripped = self.strip(path)?; - self.with_mount(path, |session| session.stat(&stripped)) + match self.with_mount(path, |session| session.stat(&stripped)) { + Err(VfsError::NotFound { .. }) if !self.mount_children(path).is_empty() => { + Ok(Stat::new(FileType::Directory, 0)) + } + served => served, + } } fn mkdir(&mut self, path: &VfsPath, recursive: bool) -> Result<(), VfsError> { diff --git a/crates/runvil-internal/vfs/src/router-tests.rs b/crates/runvil-internal/vfs/src/router-tests.rs index 4bc52fc63..1cd040358 100644 --- a/crates/runvil-internal/vfs/src/router-tests.rs +++ b/crates/runvil-internal/vfs/src/router-tests.rs @@ -4,6 +4,7 @@ use super::{Mounts, Router}; use crate::error::VfsError; use crate::handle::VfsRef; use crate::memory::MemoryBackend; +use crate::stat::FileType; use crate::test_support::{Call, TestFs, access, context, handle, path, test_origin}; use crate::traits::{Op, Vfs}; @@ -364,6 +365,84 @@ fn one_handle_serves_several_mounts_and_an_overlay_simultaneously() -> Result<() Ok(()) } +#[test] +fn a_directory_a_mount_sits_under_exists_and_lists_the_mount() -> Result<(), VfsError> { + let base = TestFs::seeded(&[("/z.txt", "z")]); + let guide = TestFs::seeded(&[("/page.md", "page")]); + let vfs = VfsRef::builder() + .mount("/", base) + .mount("/_runvil/guide", guide) + .build(); + let access = access(&vfs)?; + let names = |path: &str| -> Result, VfsError> { + Ok(access + .list(path)? + .into_iter() + .map(|entry| entry.name) + .collect()) + }; + + // No backend holds /_runvil, but a mount sits under it, so it is a + // directory. + assert!(access.exists("/_runvil")?); + assert_eq!(access.stat("/_runvil")?.file_type, FileType::Directory); + // Each level lists the mount's next segment, in name order beside + // what the serving backend holds. + assert_eq!(names("/")?, ["_runvil", "z.txt"]); + assert_eq!(names("/_runvil")?, ["guide"]); + assert_eq!(names("/_runvil/guide")?, ["page.md"]); + assert_eq!( + access.list("/_runvil")?[0].stat.file_type, + FileType::Directory + ); + // A sibling that no mount sits under is still absent. + assert!(!access.exists("/_runvil/other")?); + assert!(matches!( + access.stat("/_runvil/other"), + Err(VfsError::NotFound { .. }) + )); + Ok(()) +} + +#[test] +fn a_mount_name_the_serving_backend_already_lists_is_listed_once() -> Result<(), VfsError> { + let base = TestFs::seeded(&[ + ("/_runvil/guide/shadowed.txt", "shadowed"), + ("/_runvil/old.txt", "old"), + ]); + let guide = TestFs::seeded(&[("/page.md", "page")]); + let vfs = VfsRef::builder() + .mount("/", base) + .mount("/_runvil/guide", guide) + .build(); + let access = access(&vfs)?; + + let names: Vec = access + .list("/_runvil")? + .into_iter() + .map(|entry| entry.name) + .collect(); + assert_eq!(names, ["guide", "old.txt"]); + Ok(()) +} + +#[test] +fn an_overlay_mount_makes_its_ancestors_visible_over_a_base_that_lacks_them() -> Result<(), VfsError> +{ + let base = VfsRef::builder().mount("/", TestFs::default()).build(); + let overlay = base.overlay("/_runvil/guide", TestFs::seeded(&[("/page.md", "page")])); + let access = access(&overlay)?; + + let root: Vec = access + .list("/")? + .into_iter() + .map(|entry| entry.name) + .collect(); + assert_eq!(root, ["_runvil"]); + assert_eq!(access.read_string("/_runvil/guide/page.md")?, "page"); + Ok(()) +} + #[test] fn an_overlay_shares_the_bases_claims_table() -> Result<(), VfsError> { let base = VfsRef::builder().mount("/store", TestFs::default()).build(); diff --git a/crates/runvil-internal/vfs/src/router.rs b/crates/runvil-internal/vfs/src/router.rs index ee8644f5e..0e8301cac 100644 --- a/crates/runvil-internal/vfs/src/router.rs +++ b/crates/runvil-internal/vfs/src/router.rs @@ -7,6 +7,12 @@ //! longest-prefix mount per operation, strips the mount prefix so each //! backend sees a rooted path within its own mount, and acquires each //! backend's access lazily on first touch. +//! +//! A directory that exists only because a mount sits beneath it, such as +//! `/a` for a mount at `/a/b`, is reported as a directory: it exists, +//! stats as a directory, and lists the mount's next path segment beside +//! whatever its serving backend holds, so a listing from the root walks +//! down to every mount. #[path = "router-access.rs"] mod access; diff --git a/crates/runvil-plugin/src/lib.rs b/crates/runvil-plugin/src/lib.rs index c43729ebd..f4cb3c4d7 100644 --- a/crates/runvil-plugin/src/lib.rs +++ b/crates/runvil-plugin/src/lib.rs @@ -101,9 +101,9 @@ pub use registry::{InstallError, PackOrigin, PackRegistry}; pub use service::{HostServices, ServiceError, ServiceId, ServiceKey, TOKIO_RUNTIME}; pub use runvil_types::models::{ModelId, ModelInfo, ThinkingMode}; -pub use runvil_types::packs::{PackId, PackIdError, PackIdErrorKind}; +pub use runvil_types::names::IdError; +pub use runvil_types::packs::PackId; pub use runvil_types::tools::{ - OutputTrust, ToolCallOrigin, ToolCaller, ToolError, ToolErrorKind, ToolId, ToolIdError, - ToolIdErrorKind, ToolInfo, ToolOutput, + OutputTrust, ToolCallOrigin, ToolCaller, ToolError, ToolErrorKind, ToolId, ToolInfo, ToolOutput, }; pub use runvil_vfs::{Access, Entry, FileType, PathReason, Stat, VfsError}; diff --git a/crates/runvil/examples/greeter.rs b/crates/runvil/examples/greeter.rs index a7c7fdb2c..5bcec8984 100644 --- a/crates/runvil/examples/greeter.rs +++ b/crates/runvil/examples/greeter.rs @@ -1,36 +1,35 @@ -//! The greeter: one caller loop that parses a prompt, prepares it with a -//! canned model and a shout tool, steps the run, answers every effect it +//! The greeter: one caller loop that parses a prompt, builds its run with +//! a canned model and a shout tool, steps the run, answers every effect it //! hands back, logs every event, and prints the result. //! //! The facade suite includes this file as a module and runs `greet`, so //! the items it reaches are `pub`. +use std::collections::BTreeMap; use std::error::Error; use std::num::NonZeroU32; use std::sync::Arc; +use runvil::cancel::CancelHandle; use runvil::effect::{Effect, EffectAnswer, IssuedEffect}; use runvil::event::Event; -use runvil::model::{Completion, CompletionResult, ModelId, ModelInfo, ThinkingMode}; +use runvil::model::{Completion, CompletionResult, ModelId, ModelInfo, ModelRole, ThinkingMode}; use runvil::timestamp::Timestamp; use runvil::tools::{ToolCatalog, ToolId, ToolInfo, ToolOutput}; use runvil::vfs::perform_vfs_op; -use runvil::{Prompt, Run, RunContext, RunResult, RunSeed, Step}; +use runvil::{Prompt, Run, RunConfig, RunOutcome, RunSeed, Step}; -/// Writes a note, asks the `writer` role to reply, and shouts the reply. +/// Writes a note, asks the default model to reply, and shouts the reply. pub const GREETER: &str = concat!( "---\n", "name: greeter\n", "description: Writes a note, asks a model to reply, and shouts the reply.\n", "runvil: 0\n", - "models:\n", - " writer: {}\n", "---\n\n", "# Greeter\n\n", "## Greet\n\n", "```lua\n", "store.write('note.md', 'hello')\n", - "models.use('writer')\n", "local reply = models.infer(store.read('note.md'))\n", "return tools.call('example/text/shout', { text = reply })\n", "```\n", @@ -58,8 +57,9 @@ pub fn answer(effect: Effect) -> EffectAnswer { /// /// # Errors /// -/// Returns an error when the greeter fails to parse, prepare refuses it, -/// or the run ends without text. +/// Returns an error when the greeter fails to parse or the run ends +/// without text, as it does when its first step finds that the config +/// cannot satisfy the prompt. /// /// # Panics /// @@ -72,18 +72,18 @@ pub fn greet() -> Result> { let mut log: Vec = parse_events.clone(); let prompt = parsed?; - // 2. Build the context: run events number on from the parse events. A + // 2. Build the config: run events number on from the parse events. A // live caller draws the seed's 32 bytes from a CSPRNG; the example // fixes them. - let ctx = RunContext::new( + let ctx = RunConfig::new( "greeter", RunSeed::from_bytes([7; 32]), Timestamp::UNIX_EPOCH, ) .with_provenance_start(u32::try_from(parse_events.len())?); - // 3. Offer the shout tool, prepare under the current model, and refuse - // to run when prepare reports a gap. + // 3. Offer the shout tool and give the canned model the `default` + // role. The run's first step checks them against the prompt. let shout = ToolInfo::new( ToolId::parse("example/text/shout")?, "Returns the text in capital letters.", @@ -96,47 +96,49 @@ pub fn greet() -> Result> { NonZeroU32::new(8_192).ok_or("a context window is never zero")?, ThinkingMode::Never, ); - let (ctx, requirements) = ctx.with_tools(catalog).prepare(&prompt, Some(model)); - if let Some(refusal) = requirements.refusal() { - return Err(refusal.into()); - } + let ctx = ctx + .with_tools(catalog) + .with_models(BTreeMap::from([(ModelRole::Default, model)])); - // 4. Create the run, and keep a cancel handle for anything that may - // need to stop it. - let mut run = Run::new(Arc::new(prompt), "", ctx); - let cancel = run.cancel_handle(); + // 4. Give the config a cancel flag, keep a clone for anything that may + // need to stop the run, and create the run. + let cancel = CancelHandle::new(); + let mut run = Run::new(Arc::new(prompt), "", ctx.with_cancel(cancel.clone())); // 5. Step until Done, logging each step's events and answering each - // effect exactly once. - let result = loop { + // effect exactly once: an effect the step withdraws is dropped + // instead of performed. + let outcome = loop { match run.step() { Step::Pending { - effects, events, .. + effects, + events, + withdrawn, } => { log.extend(events); for IssuedEffect { id, effect, .. } in effects { - if run.decided() { + if withdrawn.contains(&id) { run.resume(id, EffectAnswer::Dropped); } else { run.resume(id, answer(effect)); } } } - Step::Done { result, events } => { + Step::Done { outcome, events } => { log.extend(events); - break result; + break outcome; } } }; - // 6. The tool's output is the result, and the log opens with the parse - // events. + // 6. The tool's output is the run's final text, and the log opens with + // the parse events. assert!(log.starts_with(&parse_events)); assert!(!cancel.is_cancelled()); - let RunResult::Completed(text) = result else { - return Err(format!("the greeter did not finish: {result:?}").into()); + let RunOutcome::Completed { final_text } = outcome else { + return Err(format!("the greeter did not finish: {outcome:?}").into()); }; - Ok(text) + Ok(final_text) } fn main() -> Result<(), Box> { diff --git a/crates/runvil/public-api.txt b/crates/runvil/public-api.txt index eddb3870c..150521ec0 100644 --- a/crates/runvil/public-api.txt +++ b/crates/runvil/public-api.txt @@ -1,28 +1,18 @@ #[non_exhaustive] pub enum runvil::ParseErrorKind -#[non_exhaustive] pub enum runvil::RequirementCheck #[non_exhaustive] pub enum runvil::RunErrorKind -#[non_exhaustive] pub enum runvil::RunResult -#[non_exhaustive] pub enum runvil::effect::AnswerRecord -#[non_exhaustive] pub enum runvil::effect::EffectRecord #[non_exhaustive] pub enum runvil::effect::ToolCaller -#[non_exhaustive] pub enum runvil::event::DebugMode #[non_exhaustive] pub enum runvil::event::Event #[non_exhaustive] pub enum runvil::event::ReplyOrigin #[non_exhaustive] pub enum runvil::ids::AbandonReason #[non_exhaustive] pub enum runvil::model::CompletionErrorKind #[non_exhaustive] pub enum runvil::model::CompletionResult -#[non_exhaustive] pub enum runvil::model::ModelCatalogError +#[non_exhaustive] pub enum runvil::model::ModelRole #[non_exhaustive] pub enum runvil::model::Role #[non_exhaustive] pub enum runvil::model::TemperatureError #[non_exhaustive] pub enum runvil::model::ThinkingMode -#[non_exhaustive] pub enum runvil::packs::PackIdErrorKind #[non_exhaustive] pub enum runvil::prompt::ArgType -#[non_exhaustive] pub enum runvil::prompt::ModelKeyword #[non_exhaustive] pub enum runvil::tools::OutputTrust -#[non_exhaustive] pub enum runvil::tools::ToolCatalogError -#[non_exhaustive] pub enum runvil::tools::ToolCatalogErrorKind #[non_exhaustive] pub enum runvil::tools::ToolErrorKind -#[non_exhaustive] pub enum runvil::tools::ToolIdErrorKind #[non_exhaustive] pub enum runvil::vfs::FileType #[non_exhaustive] pub enum runvil::vfs::Mode #[non_exhaustive] pub enum runvil::vfs::Op @@ -32,24 +22,13 @@ #[non_exhaustive] pub enum runvil::vfs::VfsOp #[non_exhaustive] pub enum runvil::vfs::VfsOutcome #[non_exhaustive] pub enum runvil::vfs::Writability -#[non_exhaustive] pub runvil::model::ModelCatalogError::DuplicateId { .. } #[non_exhaustive] pub runvil::model::TemperatureError::OutOfRange { .. } -#[non_exhaustive] pub runvil::tools::ToolCatalogError::DuplicateId { .. } -#[non_exhaustive] pub struct runvil::MissingService { .. } #[non_exhaustive] pub struct runvil::ParseError { /* private fields */ } #[non_exhaustive] pub struct runvil::Prompt { /* private fields */ } -#[non_exhaustive] pub struct runvil::Requirements { .. } #[non_exhaustive] pub struct runvil::RunLimits { .. } -#[non_exhaustive] pub struct runvil::SourceLocation { .. } -#[non_exhaustive] pub struct runvil::UnavailablePack { .. } -#[non_exhaustive] pub struct runvil::UnmetRequirement { .. } -#[non_exhaustive] pub struct runvil::UsablePack { .. } -#[non_exhaustive] pub struct runvil::effect::ChatAnswerRecord { .. } -#[non_exhaustive] pub struct runvil::effect::FailureRecord { .. } #[non_exhaustive] pub struct runvil::effect::IssuedEffect { .. } #[non_exhaustive] pub struct runvil::effect::PackToolSets { .. } #[non_exhaustive] pub struct runvil::effect::Round { .. } -#[non_exhaustive] pub struct runvil::effect::ToolAnswerRecord { .. } #[non_exhaustive] pub struct runvil::effect::ToolCallOrigin { .. } #[non_exhaustive] pub struct runvil::ids::Provenance { .. } #[non_exhaustive] pub struct runvil::metrics::CallMetrics { .. } @@ -64,26 +43,21 @@ #[non_exhaustive] pub struct runvil::model::Message { /* private fields */ } #[non_exhaustive] pub struct runvil::model::ModelCatalog { /* private fields */ } #[non_exhaustive] pub struct runvil::model::ModelId { /* private fields */ } -#[non_exhaustive] pub struct runvil::model::ModelIdError { /* private fields */ } #[non_exhaustive] pub struct runvil::model::ModelInfo { /* private fields */ } #[non_exhaustive] pub struct runvil::model::ModelInvocation { .. } #[non_exhaustive] pub struct runvil::model::RawExchange { /* private fields */ } #[non_exhaustive] pub struct runvil::model::ToolArguments<'a> { /* private fields */ } #[non_exhaustive] pub struct runvil::model::ToolCall { /* private fields */ } #[non_exhaustive] pub struct runvil::model::ToolSchema { /* private fields */ } -#[non_exhaustive] pub struct runvil::packs::GlobalNameError { /* private fields */ } #[non_exhaustive] pub struct runvil::packs::PackId(/* private fields */) -#[non_exhaustive] pub struct runvil::packs::PackIdError { /* private fields */ } #[non_exhaustive] pub struct runvil::prompt::ArgDecl { /* private fields */ } #[non_exhaustive] pub struct runvil::prompt::ArgsDecl { /* private fields */ } #[non_exhaustive] pub struct runvil::prompt::FileDecl { /* private fields */ } #[non_exhaustive] pub struct runvil::prompt::Frontmatter { /* private fields */ } -#[non_exhaustive] pub struct runvil::prompt::ModelRole { /* private fields */ } -#[non_exhaustive] pub struct runvil::prompt::ModelRoles { /* private fields */ } +#[non_exhaustive] pub struct runvil::prompt::RoleRequirements { /* private fields */ } #[non_exhaustive] pub struct runvil::tools::ToolCatalog { /* private fields */ } #[non_exhaustive] pub struct runvil::tools::ToolError { /* private fields */ } #[non_exhaustive] pub struct runvil::tools::ToolId(/* private fields */) -#[non_exhaustive] pub struct runvil::tools::ToolIdError { /* private fields */ } #[non_exhaustive] pub struct runvil::tools::ToolInfo { .. } #[non_exhaustive] pub struct runvil::tools::ToolOutput { /* private fields */ } #[non_exhaustive] pub struct runvil::vfs::Entry { .. } @@ -91,44 +65,30 @@ #[non_exhaustive] pub struct runvil::vfs::Origin { .. } #[non_exhaustive] pub struct runvil::vfs::RealBackend { /* private fields */ } #[non_exhaustive] pub struct runvil::vfs::Stat { .. } -auto runvil::MissingService: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::ParseError: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe auto runvil::ParseErrorKind: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::Prompt: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::RequirementCheck: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::Requirements: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::Run: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe -auto runvil::RunContext: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe -auto runvil::RunError: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe +auto runvil::RunConfig: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe auto runvil::RunErrorKind: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::RunLimits: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::RunResult: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe +auto runvil::RunOutcome: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::RunSeed: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::SourceLocation: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::Step: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe -auto runvil::UnavailablePack: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::UnmetRequirement: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::UsablePack: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::cancel::CancelHandle: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::cancel::Cancelled: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::effect::AnswerRecord: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::effect::ChatAnswerRecord: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::effect::Effect: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe auto runvil::effect::EffectAnswer: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe auto runvil::effect::EffectId: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::effect::EffectRecord: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::effect::FailureRecord: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::effect::IssuedEffect: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe auto runvil::effect::Round: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::effect::ToolAnswerRecord: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::effect::ToolCallOrigin: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::effect::ToolCaller: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::event::DebugMode: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::event::Event: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::event::ReplyOrigin: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::ids::AbandonReason: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::ids::ChainId: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::ids::ParseIdError: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe +auto runvil::ids::IdError: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::ids::Provenance: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::ids::RoundId: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::ids::TaskId: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe @@ -146,13 +106,11 @@ auto runvil::model::CompletionOptions: Freeze, RefUnwindSafe, Send, Sync, Unpin, auto runvil::model::CompletionResult: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::model::Message: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::model::ModelBinding: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::model::ModelBindings: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::model::ModelCatalog: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::model::ModelCatalogError: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::model::ModelId: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::model::ModelIdError: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::model::ModelInfo: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::model::ModelInvocation: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe +auto runvil::model::ModelRole: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::model::ParseRoleError: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::model::RawExchange: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::model::Role: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe @@ -163,33 +121,23 @@ auto runvil::model::ToolArguments: Freeze, RefUnwindSafe, Send, Sync, Unpin, Unw auto runvil::model::ToolCall: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::model::ToolSchema: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::packs::GlobalName: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::packs::GlobalNameError: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::packs::PackId: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::packs::PackIdError: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::packs::PackIdErrorKind: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::packs::PackRef: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::packs::Prelude: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe +auto runvil::packs::RosterEntry: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::prompt::ArgDecl: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::prompt::ArgType: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::prompt::ArgsDecl: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::prompt::ArgsDeclIter: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::prompt::FileDecl: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::prompt::Frontmatter: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::prompt::ModelKeyword: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::prompt::ModelRole: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::prompt::ModelRoles: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::prompt::ModelRolesIter: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::replay::Flags: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe +auto runvil::prompt::RoleRequirements: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::timestamp::Timestamp: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::tools::OutputTrust: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::tools::ToolCatalog: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::tools::ToolCatalogError: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::tools::ToolCatalogErrorKind: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::tools::ToolError: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe auto runvil::tools::ToolErrorKind: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::tools::ToolId: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::tools::ToolIdError: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -auto runvil::tools::ToolIdErrorKind: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::tools::ToolInfo: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::tools::ToolOutput: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe auto runvil::vfs::Access: !Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe @@ -218,44 +166,30 @@ auto runvil::vfs::VfsPathBuf: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSa auto runvil::vfs::VfsRef: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe auto runvil::vfs::VfsRefBuilder: Freeze, !RefUnwindSafe, Send, Sync, Unpin, !UnwindSafe auto runvil::vfs::Writability: Freeze, RefUnwindSafe, Send, Sync, Unpin, UnwindSafe -derives runvil::MissingService: Clone, Debug, Eq, PartialEq derives runvil::ParseError: Debug derives runvil::ParseErrorKind: Clone, Copy, Debug, Eq, Hash, PartialEq derives runvil::Prompt: Clone, Debug, Eq, PartialEq -derives runvil::RequirementCheck: Clone, Copy, Debug, Eq, PartialEq -derives runvil::Requirements: Clone, Debug, Default, Eq, PartialEq derives runvil::Run: Debug -derives runvil::RunContext: Debug -derives runvil::RunError: Debug +derives runvil::RunConfig: Debug derives runvil::RunErrorKind: Clone, Copy, Debug, Eq, Hash, PartialEq derives runvil::RunLimits: Clone, Copy, Debug, Default, Eq, Hash, PartialEq -derives runvil::RunResult: Debug +derives runvil::RunOutcome: Clone, Debug, Eq, PartialEq derives runvil::RunSeed: Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize -derives runvil::SourceLocation: Clone, Debug, Eq, PartialEq derives runvil::Step: Debug -derives runvil::UnavailablePack: Clone, Debug, Eq, PartialEq -derives runvil::UnmetRequirement: Clone, Debug, Eq, PartialEq -derives runvil::UsablePack: Clone, Debug, Eq, PartialEq derives runvil::cancel::CancelHandle: Clone, Debug, Default derives runvil::cancel::Cancelled: Debug -derives runvil::effect::AnswerRecord: Clone, Debug, Deserialize, PartialEq, Serialize -derives runvil::effect::ChatAnswerRecord: Clone, Debug, Deserialize, Eq, PartialEq, Serialize derives runvil::effect::Effect: Debug derives runvil::effect::EffectAnswer: Debug derives runvil::effect::EffectId: Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd -derives runvil::effect::EffectRecord: Clone, Debug, Deserialize, PartialEq, Serialize -derives runvil::effect::FailureRecord: Clone, Debug, Deserialize, Eq, PartialEq, Serialize derives runvil::effect::IssuedEffect: Debug derives runvil::effect::Round: Clone, Copy, Debug, Eq, PartialEq -derives runvil::effect::ToolAnswerRecord: Clone, Debug, Deserialize, Eq, PartialEq, Serialize derives runvil::effect::ToolCallOrigin: Clone, Debug, Deserialize, Eq, PartialEq, Serialize derives runvil::effect::ToolCaller: Clone, Copy, Debug, Deserialize, Eq, Hash, PartialEq, Serialize -derives runvil::event::DebugMode: Clone, Copy, Debug, Default, Eq, PartialEq derives runvil::event::Event: Clone, Debug, Deserialize, PartialEq, Serialize derives runvil::event::ReplyOrigin: Clone, Copy, Debug, Default, Deserialize, Eq, PartialEq, Serialize derives runvil::ids::AbandonReason: Clone, Copy, Debug, Deserialize, Eq, Hash, PartialEq, Serialize derives runvil::ids::ChainId: Clone, Debug, Deserialize, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize -derives runvil::ids::ParseIdError: Clone, Debug, Eq, PartialEq +derives runvil::ids::IdError: Clone, Debug, Eq, PartialEq derives runvil::ids::Provenance: Clone, Debug, Deserialize, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize derives runvil::ids::RoundId: Clone, Copy, Debug, Deserialize, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize derives runvil::ids::TaskId: Clone, Debug, Deserialize, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize @@ -273,13 +207,11 @@ derives runvil::model::CompletionOptions: Clone, Debug, Eq, PartialEq derives runvil::model::CompletionResult: Clone, Debug, Eq, PartialEq derives runvil::model::Message: Clone, Debug, Eq, PartialEq, Serialize derives runvil::model::ModelBinding: Clone, Debug, Eq, PartialEq -derives runvil::model::ModelBindings: Clone, Debug, Default, Eq, PartialEq derives runvil::model::ModelCatalog: Clone, Debug, Default, Eq, PartialEq -derives runvil::model::ModelCatalogError: Clone, Debug, Eq, PartialEq derives runvil::model::ModelId: Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd -derives runvil::model::ModelIdError: Clone, Debug, Eq, PartialEq derives runvil::model::ModelInfo: Clone, Debug, Eq, PartialEq derives runvil::model::ModelInvocation: Clone, Debug, Default, Eq, PartialEq +derives runvil::model::ModelRole: Clone, Copy, Debug, Deserialize, Eq, Hash, Ord, PartialEq, PartialOrd derives runvil::model::ParseRoleError: Clone, Debug, Eq, PartialEq derives runvil::model::RawExchange: Clone, Debug, Eq, PartialEq derives runvil::model::Role: Clone, Copy, Debug, Deserialize, Eq, Hash, PartialEq, Serialize @@ -290,33 +222,23 @@ derives runvil::model::ToolArguments: Clone, Copy, Debug derives runvil::model::ToolCall: Clone, Debug, Eq, PartialEq derives runvil::model::ToolSchema: Clone, Debug, Eq, PartialEq, Serialize derives runvil::packs::GlobalName: Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd -derives runvil::packs::GlobalNameError: Clone, Debug, Eq, PartialEq derives runvil::packs::PackId: Clone, Debug, Deserialize, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize -derives runvil::packs::PackIdError: Clone, Debug, Eq, PartialEq -derives runvil::packs::PackIdErrorKind: Clone, Copy, Debug, Eq, PartialEq derives runvil::packs::PackRef: Clone, Debug, Deserialize, Eq, PartialEq derives runvil::packs::Prelude: Clone, Debug, Eq, PartialEq +derives runvil::packs::RosterEntry: Clone, Debug, Eq, PartialEq derives runvil::prompt::ArgDecl: Clone, Debug, Deserialize, Eq, PartialEq derives runvil::prompt::ArgType: Clone, Copy, Debug, Deserialize, Eq, PartialEq derives runvil::prompt::ArgsDecl: Clone, Debug, Default, Deserialize, Eq, PartialEq derives runvil::prompt::ArgsDeclIter: Clone, Debug derives runvil::prompt::FileDecl: Clone, Debug, Deserialize, Eq, PartialEq derives runvil::prompt::Frontmatter: Clone, Debug, Deserialize, Eq, PartialEq -derives runvil::prompt::ModelKeyword: Clone, Copy, Debug, Deserialize, Eq, Ord, PartialEq, PartialOrd -derives runvil::prompt::ModelRole: Clone, Debug, Deserialize, Eq, PartialEq -derives runvil::prompt::ModelRoles: Clone, Debug, Default, Deserialize, Eq, PartialEq -derives runvil::prompt::ModelRolesIter: Clone, Debug -derives runvil::replay::Flags: Clone, Copy, Debug, Default, Deserialize, Eq, Hash, PartialEq, Serialize +derives runvil::prompt::RoleRequirements: Clone, Copy, Debug, Deserialize, Eq, PartialEq derives runvil::timestamp::Timestamp: Clone, Copy, Debug, Default, Deserialize, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize derives runvil::tools::OutputTrust: Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize derives runvil::tools::ToolCatalog: Clone, Debug, Default -derives runvil::tools::ToolCatalogError: Clone, Debug, Eq, PartialEq -derives runvil::tools::ToolCatalogErrorKind: Clone, Copy, Debug, Eq, PartialEq derives runvil::tools::ToolError: Debug derives runvil::tools::ToolErrorKind: Clone, Copy, Debug, Eq, PartialEq derives runvil::tools::ToolId: Clone, Debug, Deserialize, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize -derives runvil::tools::ToolIdError: Clone, Debug, Eq, PartialEq -derives runvil::tools::ToolIdErrorKind: Clone, Copy, Debug, Eq, PartialEq derives runvil::tools::ToolInfo: Clone, Debug, Deserialize, Eq, PartialEq, Serialize derives runvil::tools::ToolOutput: Clone, Debug, Eq, PartialEq derives runvil::vfs::Access: Debug @@ -345,47 +267,32 @@ derives runvil::vfs::VfsPathBuf: Clone, Debug, Eq, Hash, Ord, PartialEq, Partial derives runvil::vfs::VfsRef: Clone, Debug, Default derives runvil::vfs::VfsRefBuilder: Debug derives runvil::vfs::Writability: Clone, Copy, Debug, Eq, Hash, PartialEq -impl core::convert::From<&runvil::model::Completion> for runvil::effect::ChatAnswerRecord impl core::convert::From for runvil::ids::TaskId impl core::convert::From for runvil::vfs::VfsPathBuf impl core::error::Error for runvil::ParseError -impl core::error::Error for runvil::RunError -impl core::error::Error for runvil::ids::ParseIdError +impl core::error::Error for runvil::ids::IdError impl core::error::Error for runvil::model::CompletionError -impl core::error::Error for runvil::model::ModelCatalogError -impl core::error::Error for runvil::model::ModelIdError impl core::error::Error for runvil::model::ParseRoleError impl core::error::Error for runvil::model::TemperatureError -impl core::error::Error for runvil::packs::GlobalNameError -impl core::error::Error for runvil::packs::PackIdError -impl core::error::Error for runvil::tools::ToolCatalogError impl core::error::Error for runvil::tools::ToolError -impl core::error::Error for runvil::tools::ToolIdError impl core::error::Error for runvil::vfs::ParsePathReasonError impl core::error::Error for runvil::vfs::VfsError impl core::fmt::Display for runvil::ParseError -impl core::fmt::Display for runvil::RunError impl core::fmt::Display for runvil::RunSeed impl core::fmt::Display for runvil::effect::EffectId impl core::fmt::Display for runvil::ids::ChainId -impl core::fmt::Display for runvil::ids::ParseIdError +impl core::fmt::Display for runvil::ids::IdError impl core::fmt::Display for runvil::ids::TaskId impl core::fmt::Display for runvil::model::CompletionError -impl core::fmt::Display for runvil::model::ModelCatalogError -impl core::fmt::Display for runvil::model::ModelIdError impl core::fmt::Display for runvil::model::ParseRoleError impl core::fmt::Display for runvil::model::Role impl core::fmt::Display for runvil::model::TemperatureError impl core::fmt::Display for runvil::packs::GlobalName -impl core::fmt::Display for runvil::packs::GlobalNameError impl core::fmt::Display for runvil::packs::PackId -impl core::fmt::Display for runvil::packs::PackIdError impl core::fmt::Display for runvil::prompt::ArgType impl core::fmt::Display for runvil::timestamp::Timestamp -impl core::fmt::Display for runvil::tools::ToolCatalogError impl core::fmt::Display for runvil::tools::ToolError impl core::fmt::Display for runvil::tools::ToolId -impl core::fmt::Display for runvil::tools::ToolIdError impl core::fmt::Display for runvil::vfs::ParsePathReasonError impl core::fmt::Display for runvil::vfs::PathReason impl core::fmt::Display for runvil::vfs::VfsError @@ -394,23 +301,19 @@ impl core::fmt::Display for runvil::vfs::VfsPathBuf impl core::future::future::Future for runvil::cancel::Cancelled impl core::future::future::Future for runvil::cancel::Cancelled { type Output = () } impl core::iter::traits::exact_size::ExactSizeIterator for runvil::prompt::ArgsDeclIter<'_> -impl core::iter::traits::exact_size::ExactSizeIterator for runvil::prompt::ModelRolesIter<'_> -impl core::ops::bit::BitOr for runvil::replay::Flags -impl core::ops::bit::BitOr for runvil::replay::Flags { type Output = runvil::replay::Flags } -impl core::ops::bit::BitOrAssign for runvil::replay::Flags impl core::ops::drop::Drop for runvil::vfs::Access impl core::str::traits::FromStr for runvil::ids::ChainId -impl core::str::traits::FromStr for runvil::ids::ChainId { type Err = runvil::ids::ParseIdError } +impl core::str::traits::FromStr for runvil::ids::ChainId { type Err = runvil::ids::IdError } impl core::str::traits::FromStr for runvil::ids::TaskId -impl core::str::traits::FromStr for runvil::ids::TaskId { type Err = runvil::ids::ParseIdError } +impl core::str::traits::FromStr for runvil::ids::TaskId { type Err = runvil::ids::IdError } impl core::str::traits::FromStr for runvil::model::Role impl core::str::traits::FromStr for runvil::model::Role { type Err = runvil::model::ParseRoleError } impl core::str::traits::FromStr for runvil::packs::GlobalName -impl core::str::traits::FromStr for runvil::packs::GlobalName { type Err = runvil::packs::GlobalNameError } +impl core::str::traits::FromStr for runvil::packs::GlobalName { type Err = runvil::ids::IdError } impl core::str::traits::FromStr for runvil::packs::PackId -impl core::str::traits::FromStr for runvil::packs::PackId { type Err = runvil::packs::PackIdError } +impl core::str::traits::FromStr for runvil::packs::PackId { type Err = runvil::ids::IdError } impl core::str::traits::FromStr for runvil::tools::ToolId -impl core::str::traits::FromStr for runvil::tools::ToolId { type Err = runvil::tools::ToolIdError } +impl core::str::traits::FromStr for runvil::tools::ToolId { type Err = runvil::ids::IdError } impl core::str::traits::FromStr for runvil::vfs::PathReason impl core::str::traits::FromStr for runvil::vfs::PathReason { type Err = runvil::vfs::ParsePathReasonError } impl runvil::vfs::Policy for runvil::vfs::AllowAll @@ -421,15 +324,10 @@ impl runvil::vfs::Vfs for runvil::vfs::VfsRef impl<'a> core::iter::traits::collect::IntoIterator for &'a runvil::prompt::ArgsDecl impl<'a> core::iter::traits::collect::IntoIterator for &'a runvil::prompt::ArgsDecl { type IntoIter = runvil::prompt::ArgsDeclIter<'a> } impl<'a> core::iter::traits::collect::IntoIterator for &'a runvil::prompt::ArgsDecl { type Item = (&'a str, &'a runvil::prompt::ArgDecl) } -impl<'a> core::iter::traits::collect::IntoIterator for &'a runvil::prompt::ModelRoles -impl<'a> core::iter::traits::collect::IntoIterator for &'a runvil::prompt::ModelRoles { type IntoIter = runvil::prompt::ModelRolesIter<'a> } -impl<'a> core::iter::traits::collect::IntoIterator for &'a runvil::prompt::ModelRoles { type Item = (&'a str, &'a runvil::prompt::ModelRole) } impl<'a> core::iter::traits::iterator::Iterator for runvil::model::ArgumentNames<'a> impl<'a> core::iter::traits::iterator::Iterator for runvil::model::ArgumentNames<'a> { type Item = &'a str } impl<'a> core::iter::traits::iterator::Iterator for runvil::prompt::ArgsDeclIter<'a> impl<'a> core::iter::traits::iterator::Iterator for runvil::prompt::ArgsDeclIter<'a> { type Item = (&'a str, &'a runvil::prompt::ArgDecl) } -impl<'a> core::iter::traits::iterator::Iterator for runvil::prompt::ModelRolesIter<'a> -impl<'a> core::iter::traits::iterator::Iterator for runvil::prompt::ModelRolesIter<'a> { type Item = (&'a str, &'a runvil::prompt::ModelRole) } impl<'a> runvil::vfs::OpEvent<'a> impl<'de, T> serde_core::de::Deserialize<'de> for runvil::effect::PackToolSets where T: serde_core::de::Deserialize<'de> impl core::clone::Clone for runvil::effect::PackToolSets @@ -444,25 +342,24 @@ impl core::panic::unwind_safe::RefUnwindSafe for runvil::effect::PackToolSets impl core::panic::unwind_safe::UnwindSafe for runvil::effect::PackToolSets where alloc::vec::Vec: core::panic::unwind_safe::UnwindSafe + core::panic::unwind_safe::UnwindSafe impl runvil::effect::PackToolSets impl serde_core::ser::Serialize for runvil::effect::PackToolSets where T: serde_core::ser::Serialize +pub const fn runvil::RunOutcome::as_str(&self) -> &'static str pub const fn runvil::RunSeed::as_bytes(&self) -> &[u8; 32] pub const fn runvil::RunSeed::from_bytes(bytes: [u8; 32]) -> runvil::RunSeed pub const fn runvil::effect::EffectId::get(self) -> u64 pub const fn runvil::ids::RoundId::get(self) -> u64 pub const fn runvil::ids::RoundId::new(id: u64) -> Self -pub const fn runvil::replay::Flags::bits(self) -> u32 -pub const fn runvil::replay::Flags::contains(self, other: runvil::replay::Flags) -> bool -pub const fn runvil::replay::Flags::from_bits(bits: u32) -> Self -pub const fn runvil::replay::Flags::is_empty(self) -> bool +pub const fn runvil::model::ModelRole::as_str(self) -> &'static str pub const fn runvil::timestamp::Timestamp::from_unix_millis(millis: i64) -> Self pub const fn runvil::timestamp::Timestamp::unix_millis(self) -> i64 pub const runvil::model::ModelId::GATEWAY: &'static str +pub const runvil::model::ModelRole::ALL: [runvil::model::ModelRole; 5] pub const runvil::model::Role::ALL: &'static [runvil::model::Role] -pub const runvil::replay::Flags::EMPTY: runvil::replay::Flags pub const runvil::timestamp::Timestamp::UNIX_EPOCH: runvil::timestamp::Timestamp +pub enum runvil::RunOutcome pub enum runvil::Step pub enum runvil::effect::Effect pub enum runvil::effect::EffectAnswer -pub fn runvil::MissingService::new(pack: runvil::packs::PackId, service: impl core::convert::Into) -> runvil::MissingService +pub enum runvil::packs::RosterEntry pub fn runvil::ParseError::cause(&self) -> core::option::Option<&dyn core::error::Error + core::marker::Send + core::marker::Sync + 'static> pub fn runvil::ParseError::column(&self) -> core::option::Option pub fn runvil::ParseError::kind(&self) -> runvil::ParseErrorKind @@ -472,47 +369,29 @@ pub fn runvil::ParseError::span(&self) -> core::option::Option &runvil::prompt::Frontmatter pub fn runvil::Prompt::parse(input: &str, execution: &str) -> (core::result::Result, alloc::vec::Vec) pub fn runvil::Prompt::title(&self) -> &str -pub fn runvil::Requirements::is_satisfied(&self) -> bool -pub fn runvil::Requirements::merge(&mut self, other: runvil::Requirements) -pub fn runvil::Requirements::notice(&self) -> alloc::string::String -pub fn runvil::Requirements::refusal(&self) -> core::option::Option -pub fn runvil::Run::cancel(&mut self) -pub fn runvil::Run::cancel_handle(&self) -> runvil::cancel::CancelHandle -pub fn runvil::Run::decided(&self) -> bool -pub fn runvil::Run::new(prompt: alloc::sync::Arc, args: &str, config: runvil::RunContext) -> runvil::Run +pub fn runvil::Run::new(prompt: alloc::sync::Arc, args: &str, config: runvil::RunConfig) -> runvil::Run pub fn runvil::Run::resume(&mut self, id: runvil::effect::EffectId, answer: runvil::effect::EffectAnswer) pub fn runvil::Run::step(&mut self) -> runvil::Step -pub fn runvil::RunContext::flags(&self) -> runvil::replay::Flags -pub fn runvil::RunContext::model_bindings(&self) -> &runvil::model::ModelBindings -pub fn runvil::RunContext::new(name: impl core::convert::Into, seed: runvil::RunSeed, started_at: runvil::timestamp::Timestamp) -> runvil::RunContext -pub fn runvil::RunContext::prepare(self, prompt: &runvil::Prompt, model: core::option::Option) -> (runvil::RunContext, runvil::Requirements) -pub fn runvil::RunContext::seed(&self) -> runvil::RunSeed -pub fn runvil::RunContext::started_at(&self) -> runvil::timestamp::Timestamp -pub fn runvil::RunContext::tools(&self) -> &runvil::tools::ToolCatalog -pub fn runvil::RunContext::vfs(&self) -> &runvil::vfs::VfsRef -pub fn runvil::RunContext::with_cancel(self, handle: runvil::cancel::CancelHandle) -> runvil::RunContext -pub fn runvil::RunContext::with_flags(self, flags: runvil::replay::Flags) -> runvil::RunContext -pub fn runvil::RunContext::with_limits(self, limits: runvil::RunLimits) -> runvil::RunContext -pub fn runvil::RunContext::with_packs(self, usable: alloc::vec::Vec, unavailable: alloc::vec::Vec) -> runvil::RunContext -pub fn runvil::RunContext::with_preludes(self, preludes: alloc::vec::Vec) -> runvil::RunContext -pub fn runvil::RunContext::with_provenance_start(self, start: u32) -> runvil::RunContext -pub fn runvil::RunContext::with_report_debug(self, mode: runvil::event::DebugMode) -> runvil::RunContext -pub fn runvil::RunContext::with_tools(self, tools: runvil::tools::ToolCatalog) -> runvil::RunContext -pub fn runvil::RunContext::with_ui(self, snapshot: serde_json::value::Value) -> runvil::RunContext -pub fn runvil::RunContext::with_vfs(self, vfs: runvil::vfs::VfsRef) -> runvil::RunContext -pub fn runvil::RunError::is_retryable(&self) -> bool -pub fn runvil::RunError::kind(&self) -> runvil::RunErrorKind -pub fn runvil::RunError::location(&self) -> core::option::Option -pub fn runvil::RunError::originating_error(&self) -> core::option::Option<&dyn core::error::Error + core::marker::Send + core::marker::Sync + 'static> -pub fn runvil::UnavailablePack::new(pack: runvil::packs::PackId, reason: impl core::convert::Into) -> runvil::UnavailablePack -pub fn runvil::UsablePack::new(pack: runvil::packs::PackId, vendor: impl core::convert::Into) -> runvil::UsablePack +pub fn runvil::RunConfig::new(name: impl core::convert::Into, seed: runvil::RunSeed, started_at: runvil::timestamp::Timestamp) -> runvil::RunConfig +pub fn runvil::RunConfig::with_cancel(self, handle: runvil::cancel::CancelHandle) -> runvil::RunConfig +pub fn runvil::RunConfig::with_limits(self, limits: runvil::RunLimits) -> runvil::RunConfig +pub fn runvil::RunConfig::with_models(self, models: alloc::collections::btree::map::BTreeMap) -> runvil::RunConfig +pub fn runvil::RunConfig::with_packs(self, roster: impl core::iter::traits::collect::IntoIterator) -> runvil::RunConfig +pub fn runvil::RunConfig::with_preludes(self, preludes: alloc::vec::Vec) -> runvil::RunConfig +pub fn runvil::RunConfig::with_provenance_start(self, start: u32) -> runvil::RunConfig +pub fn runvil::RunConfig::with_report_debug(self, report: bool) -> runvil::RunConfig +pub fn runvil::RunConfig::with_tools(self, tools: runvil::tools::ToolCatalog) -> runvil::RunConfig +pub fn runvil::RunConfig::with_ui(self, snapshot: serde_json::value::Value) -> runvil::RunConfig +pub fn runvil::RunConfig::with_vfs(self, vfs: runvil::vfs::VfsRef) -> runvil::RunConfig +pub fn runvil::RunErrorKind::as_str(self) -> &'static str +pub fn runvil::RunErrorKind::from_tag(tag: &str) -> core::option::Option pub fn runvil::cancel::CancelHandle::cancel(&self) pub fn runvil::cancel::CancelHandle::cancelled(&self) -> runvil::cancel::Cancelled pub fn runvil::cancel::CancelHandle::child(&self) -> runvil::cancel::CancelHandle pub fn runvil::cancel::CancelHandle::is_cancelled(&self) -> bool pub fn runvil::cancel::CancelHandle::new() -> Self -pub fn runvil::effect::Effect::record(&self) -> runvil::effect::EffectRecord -pub fn runvil::effect::EffectAnswer::record(&self) -> runvil::effect::AnswerRecord +pub fn runvil::effect::Effect::record(&self) -> serde_json::value::Value +pub fn runvil::effect::EffectAnswer::record(&self) -> serde_json::value::Value pub fn runvil::effect::PackToolSets::new(pack: runvil::packs::PackId, static_set: alloc::vec::Vec, dynamic_set: alloc::vec::Vec) -> runvil::effect::PackToolSets pub fn runvil::effect::Round::new(id: runvil::ids::RoundId, origin: runvil::event::ReplyOrigin) -> runvil::effect::Round pub fn runvil::effect::ToolCallOrigin::new(execution: impl core::convert::Into, section: impl core::convert::Into, caller: runvil::effect::ToolCaller) -> runvil::effect::ToolCallOrigin @@ -525,7 +404,6 @@ pub fn runvil::ids::AbandonReason::why(self) -> &'static str pub fn runvil::ids::ChainId::child(&self, index: u32) -> Self pub fn runvil::ids::ChainId::entry(&self, index: u32) -> alloc::string::String pub fn runvil::ids::ChainId::root() -> Self -pub fn runvil::ids::ParseIdError::input(&self) -> &str pub fn runvil::ids::Provenance::new(task: runvil::ids::TaskId, seq: u32) -> runvil::ids::Provenance pub fn runvil::metrics::CallMetrics::new() -> runvil::metrics::CallMetrics pub fn runvil::metrics::CallMetrics::with_client(self, client: core::option::Option) -> runvil::metrics::CallMetrics @@ -568,7 +446,6 @@ pub fn runvil::model::CompletionError::context_overflow(prompt_tokens: core::opt pub fn runvil::model::CompletionError::detail(&self) -> core::option::Option<&str> pub fn runvil::model::CompletionError::empty_reply(reasoning_present: bool) -> runvil::model::CompletionError pub fn runvil::model::CompletionError::finish_reason(&self) -> core::option::Option<&str> -pub fn runvil::model::CompletionError::is_retryable(&self) -> bool pub fn runvil::model::CompletionError::kind(&self) -> runvil::model::CompletionErrorKind pub fn runvil::model::CompletionError::message(&self) -> &str pub fn runvil::model::CompletionError::new(kind: runvil::model::CompletionErrorKind, message: impl core::convert::Into) -> runvil::model::CompletionError @@ -595,7 +472,6 @@ pub fn runvil::model::Message::role(&self) -> runvil::model::Role pub fn runvil::model::Message::tool(tool_call_id: impl core::convert::Into, content: impl core::convert::Into) -> runvil::model::Message pub fn runvil::model::Message::user(content: impl core::convert::Into) -> runvil::model::Message pub fn runvil::model::ModelBinding::alias(&self) -> &str -pub fn runvil::model::ModelBinding::capabilities(&self) -> &[alloc::string::String] pub fn runvil::model::ModelBinding::completion_options(&self) -> runvil::model::CompletionOptions pub fn runvil::model::ModelBinding::context(&self) -> core::num::nonzero::NonZeroU32 pub fn runvil::model::ModelBinding::description(&self) -> &str @@ -604,23 +480,17 @@ pub fn runvil::model::ModelBinding::invocation(&self) -> &runvil::model::ModelIn pub fn runvil::model::ModelBinding::new(alias: impl core::convert::Into, description: impl core::convert::Into, id: runvil::model::ModelId, invocation: runvil::model::ModelInvocation, context: core::num::nonzero::NonZeroU32) -> Self pub fn runvil::model::ModelBinding::provider(&self) -> core::option::Option<&str> pub fn runvil::model::ModelBinding::render_info(&self) -> runvil::model::ModelInfo -pub fn runvil::model::ModelBinding::with_capabilities(self, capabilities: alloc::vec::Vec) -> Self pub fn runvil::model::ModelBinding::with_catalog_info(self, info: runvil::model::ModelInfo) -> Self pub fn runvil::model::ModelBinding::with_invocation(self, invocation: runvil::model::ModelInvocation) -> Self pub fn runvil::model::ModelBinding::with_provider(self, provider: impl core::convert::Into) -> Self -pub fn runvil::model::ModelBindings::is_empty(&self) -> bool -pub fn runvil::model::ModelBindings::len(&self) -> usize -pub fn runvil::model::ModelBindings::model(&self, id: &runvil::model::ModelId) -> core::option::Option<&runvil::model::ModelInfo> -pub fn runvil::model::ModelBindings::resolve(&self, label: &str) -> core::option::Option<&runvil::model::ModelInfo> -pub fn runvil::model::ModelBindings::role_id(&self, label: &str) -> core::option::Option<&runvil::model::ModelId> pub fn runvil::model::ModelCatalog::contains(&self, id: &runvil::model::ModelId) -> bool pub fn runvil::model::ModelCatalog::empty() -> Self pub fn runvil::model::ModelCatalog::get(&self, id: &runvil::model::ModelId) -> core::option::Option<&runvil::model::ModelInfo> pub fn runvil::model::ModelCatalog::models(&self) -> &[runvil::model::ModelInfo] -pub fn runvil::model::ModelCatalog::new(models: impl core::iter::traits::collect::IntoIterator) -> core::result::Result -pub fn runvil::model::ModelId::gateway(name: impl core::convert::Into) -> core::result::Result +pub fn runvil::model::ModelCatalog::new(models: impl core::iter::traits::collect::IntoIterator) -> core::result::Result +pub fn runvil::model::ModelId::gateway(name: impl core::convert::Into) -> core::result::Result pub fn runvil::model::ModelId::name(&self) -> &str -pub fn runvil::model::ModelId::new(server: impl core::convert::Into, name: impl core::convert::Into) -> core::result::Result +pub fn runvil::model::ModelId::new(server: impl core::convert::Into, name: impl core::convert::Into) -> core::result::Result pub fn runvil::model::ModelInfo::context(&self) -> core::num::nonzero::NonZeroU32 pub fn runvil::model::ModelInfo::description(&self) -> &str pub fn runvil::model::ModelInfo::id(&self) -> &runvil::model::ModelId @@ -650,11 +520,10 @@ pub fn runvil::model::ToolCall::name(&self) -> &str pub fn runvil::model::ToolSchema::description(&self) -> &str pub fn runvil::model::ToolSchema::name(&self) -> &str pub fn runvil::model::ToolSchema::parameters(&self) -> &serde_json::value::Value -pub fn runvil::packs::GlobalName::parse(s: &str) -> core::result::Result +pub fn runvil::packs::GlobalName::parse(s: &str) -> core::result::Result pub fn runvil::packs::PackId::contains(&self, tool: &runvil::tools::ToolId) -> bool pub fn runvil::packs::PackId::is_reserved(&self) -> bool -pub fn runvil::packs::PackId::parse(name: &str) -> core::result::Result -pub fn runvil::packs::PackIdError::kind(&self) -> runvil::packs::PackIdErrorKind +pub fn runvil::packs::PackId::parse(name: &str) -> core::result::Result pub fn runvil::packs::PackRef::id(&self) -> &runvil::packs::PackId pub fn runvil::packs::PackRef::vendor(&self) -> &str pub fn runvil::packs::Prelude::new(pack: runvil::packs::PackId, source: impl core::convert::Into) -> runvil::packs::Prelude @@ -675,26 +544,17 @@ pub fn runvil::prompt::Frontmatter::args(&self) -> &runvil::prompt::ArgsDecl pub fn runvil::prompt::Frontmatter::description(&self) -> &str pub fn runvil::prompt::Frontmatter::input(&self) -> core::option::Option<&runvil::prompt::FileDecl> pub fn runvil::prompt::Frontmatter::max_tool_iterations(&self) -> core::option::Option -pub fn runvil::prompt::Frontmatter::models(&self) -> &runvil::prompt::ModelRoles pub fn runvil::prompt::Frontmatter::name(&self) -> &str pub fn runvil::prompt::Frontmatter::output(&self) -> core::option::Option<&runvil::prompt::FileDecl> +pub fn runvil::prompt::Frontmatter::role_requirements(&self, role: runvil::model::ModelRole) -> core::option::Option<&runvil::prompt::RoleRequirements> pub fn runvil::prompt::Frontmatter::runvil(&self) -> core::option::Option pub fn runvil::prompt::Frontmatter::uses(&self) -> &[runvil::packs::PackRef] -pub fn runvil::prompt::ModelRole::description(&self) -> core::option::Option<&str> -pub fn runvil::prompt::ModelRole::keywords(&self) -> &[runvil::prompt::ModelKeyword] -pub fn runvil::prompt::ModelRole::min_context(&self) -> core::option::Option -pub fn runvil::prompt::ModelRoles::get(&self, label: &str) -> core::option::Option<&runvil::prompt::ModelRole> -pub fn runvil::prompt::ModelRoles::is_empty(&self) -> bool -pub fn runvil::prompt::ModelRoles::iter(&self) -> runvil::prompt::ModelRolesIter<'_> -pub fn runvil::prompt::ModelRoles::len(&self) -> usize +pub fn runvil::prompt::RoleRequirements::min_context(&self) -> core::option::Option +pub fn runvil::prompt::RoleRequirements::thinking(&self) -> core::option::Option pub fn runvil::timestamp::Timestamp::to_rfc3339(self) -> alloc::string::String pub fn runvil::tools::ToolCatalog::get(&self, id: &runvil::tools::ToolId) -> core::option::Option<&runvil::tools::ToolInfo> -pub fn runvil::tools::ToolCatalog::new(tools: impl core::iter::traits::collect::IntoIterator) -> core::result::Result +pub fn runvil::tools::ToolCatalog::new(tools: impl core::iter::traits::collect::IntoIterator) -> core::result::Result pub fn runvil::tools::ToolCatalog::tools(&self) -> &[runvil::tools::ToolInfo] -pub fn runvil::tools::ToolCatalogError::duplicate_id(&self) -> core::option::Option<&runvil::tools::ToolId> -pub fn runvil::tools::ToolCatalogError::kind(&self) -> runvil::tools::ToolCatalogErrorKind -pub fn runvil::tools::ToolError::is_cancelled(&self) -> bool -pub fn runvil::tools::ToolError::is_retryable(&self) -> bool pub fn runvil::tools::ToolError::kind(&self) -> runvil::tools::ToolErrorKind pub fn runvil::tools::ToolError::message(text: impl core::convert::Into) -> runvil::tools::ToolError pub fn runvil::tools::ToolError::new(kind: runvil::tools::ToolErrorKind, text: impl core::convert::Into) -> runvil::tools::ToolError @@ -703,9 +563,9 @@ pub fn runvil::tools::ToolError::with_source(self, source: impl core::convert::I pub fn runvil::tools::ToolErrorKind::tag(self) -> &'static str pub fn runvil::tools::ToolId::name(&self) -> &str pub fn runvil::tools::ToolId::pack(&self) -> runvil::packs::PackId -pub fn runvil::tools::ToolId::parse(id: &str) -> core::result::Result -pub fn runvil::tools::ToolIdError::kind(&self) -> runvil::tools::ToolIdErrorKind +pub fn runvil::tools::ToolId::parse(id: &str) -> core::result::Result pub fn runvil::tools::ToolInfo::describes_tools(self, describes: bool) -> runvil::tools::ToolInfo +pub fn runvil::tools::ToolInfo::model_only_call(self, model_only: bool) -> runvil::tools::ToolInfo pub fn runvil::tools::ToolInfo::new(id: runvil::tools::ToolId, description: impl core::convert::Into, parameters_schema: serde_json::value::Value) -> runvil::tools::ToolInfo pub fn runvil::tools::ToolInfo::structured(self, structured: bool) -> runvil::tools::ToolInfo pub fn runvil::tools::ToolInfo::survives_stop(self, survives: bool) -> runvil::tools::ToolInfo @@ -722,6 +582,7 @@ pub fn runvil::vfs::Access::glob(&self, pattern: &str) -> core::result::Result core::result::Result, runvil::vfs::VfsError> pub fn runvil::vfs::Access::mkdir(&self, path: &str, recursive: bool) -> core::result::Result<(), runvil::vfs::VfsError> pub fn runvil::vfs::Access::read(&self, path: &str) -> core::result::Result, runvil::vfs::VfsError> +pub fn runvil::vfs::Access::read_range(&self, path: &str, offset: u64, len: u64) -> core::result::Result, runvil::vfs::VfsError> pub fn runvil::vfs::Access::read_string(&self, path: &str) -> core::result::Result pub fn runvil::vfs::Access::remove(&self, path: &str, recursive: bool) -> core::result::Result pub fn runvil::vfs::Access::rename(&self, from: &str, to: &str) -> core::result::Result<(), runvil::vfs::VfsError> @@ -795,79 +656,42 @@ pub mod runvil::metrics pub mod runvil::model pub mod runvil::packs pub mod runvil::prompt -pub mod runvil::replay pub mod runvil::timestamp pub mod runvil::tools pub mod runvil::vfs -pub runvil::MissingService::pack: runvil::packs::PackId -pub runvil::MissingService::service: alloc::string::String pub runvil::ParseErrorKind::Fence pub runvil::ParseErrorKind::Frontmatter pub runvil::ParseErrorKind::Internal pub runvil::ParseErrorKind::List pub runvil::ParseErrorKind::Lua pub runvil::ParseErrorKind::Structure -pub runvil::RequirementCheck::ContextMinimum -pub runvil::RequirementCheck::HardKeyword -pub runvil::Requirements::missing_packs: alloc::vec::Vec -pub runvil::Requirements::missing_services: alloc::vec::Vec -pub runvil::Requirements::reserved: alloc::collections::btree::set::BTreeSet -pub runvil::Requirements::unavailable_packs: alloc::vec::Vec -pub runvil::Requirements::unmet_models: alloc::vec::Vec pub runvil::RunErrorKind::Binding pub runvil::RunErrorKind::Completion pub runvil::RunErrorKind::ContextExhausted -pub runvil::RunErrorKind::Determinism pub runvil::RunErrorKind::Internal pub runvil::RunErrorKind::Lua pub runvil::RunErrorKind::Parse pub runvil::RunErrorKind::Quota pub runvil::RunErrorKind::RequirementsUnmet pub runvil::RunErrorKind::Tool -pub runvil::RunErrorKind::Version pub runvil::RunErrorKind::Vfs pub runvil::RunLimits::lua_log_events: core::num::nonzero::NonZeroU32 pub runvil::RunLimits::lua_memory_bytes: core::num::nonzero::NonZeroUsize pub runvil::RunLimits::max_concurrency: core::num::nonzero::NonZeroUsize pub runvil::RunLimits::max_tool_iterations: core::num::nonzero::NonZeroU32 -pub runvil::RunResult::Cancelled -pub runvil::RunResult::Completed(..) -pub runvil::RunResult::Completed::0: alloc::string::String -pub runvil::RunResult::Failure(..) -pub runvil::RunResult::Failure::0: runvil::RunError -pub runvil::SourceLocation::column: core::option::Option -pub runvil::SourceLocation::line: core::option::Option -pub runvil::SourceLocation::path: alloc::string::String -pub runvil::SourceLocation::span: core::option::Option> +pub runvil::RunOutcome::Cancelled +pub runvil::RunOutcome::Completed { .. } +pub runvil::RunOutcome::Completed::final_text: alloc::string::String +pub runvil::RunOutcome::Failed { .. } +pub runvil::RunOutcome::Failed::kind: runvil::RunErrorKind +pub runvil::RunOutcome::Failed::message: alloc::string::String pub runvil::Step::Done { .. } pub runvil::Step::Done::events: alloc::vec::Vec -pub runvil::Step::Done::result: runvil::RunResult +pub runvil::Step::Done::outcome: runvil::RunOutcome pub runvil::Step::Pending { .. } pub runvil::Step::Pending::effects: alloc::vec::Vec pub runvil::Step::Pending::events: alloc::vec::Vec pub runvil::Step::Pending::withdrawn: alloc::vec::Vec -pub runvil::UnavailablePack::pack: runvil::packs::PackId -pub runvil::UnavailablePack::reason: alloc::string::String -pub runvil::UnmetRequirement::actual: alloc::string::String -pub runvil::UnmetRequirement::check: runvil::RequirementCheck -pub runvil::UnmetRequirement::required: alloc::string::String -pub runvil::UnmetRequirement::role: alloc::string::String -pub runvil::UsablePack::pack: runvil::packs::PackId -pub runvil::UsablePack::vendor: alloc::string::String -pub runvil::effect::AnswerRecord::Chat(..) -pub runvil::effect::AnswerRecord::Chat::0: core::result::Result -pub runvil::effect::AnswerRecord::Dropped -pub runvil::effect::AnswerRecord::Render(..) -pub runvil::effect::AnswerRecord::Render::0: alloc::vec::Vec> -pub runvil::effect::AnswerRecord::Timer -pub runvil::effect::AnswerRecord::ToolCall(..) -pub runvil::effect::AnswerRecord::ToolCall::0: core::result::Result -pub runvil::effect::AnswerRecord::Vfs(..) -pub runvil::effect::AnswerRecord::Vfs::0: core::result::Result -pub runvil::effect::ChatAnswerRecord::finish_reason: core::option::Option -pub runvil::effect::ChatAnswerRecord::model: alloc::string::String -pub runvil::effect::ChatAnswerRecord::reply: core::option::Option -pub runvil::effect::ChatAnswerRecord::tool_calls: alloc::vec::Vec pub runvil::effect::Effect::Chat { .. } pub runvil::effect::Effect::Chat::after: core::option::Option<(runvil::ids::RoundId, u64)> pub runvil::effect::Effect::Chat::binding: runvil::model::ModelBinding @@ -900,31 +724,6 @@ pub runvil::effect::EffectAnswer::ToolCall(..) pub runvil::effect::EffectAnswer::ToolCall::0: core::result::Result pub runvil::effect::EffectAnswer::Vfs(..) pub runvil::effect::EffectAnswer::Vfs::0: core::result::Result -pub runvil::effect::EffectRecord::Chat { .. } -pub runvil::effect::EffectRecord::Chat::after: core::option::Option -pub runvil::effect::EffectRecord::Chat::alias: alloc::string::String -pub runvil::effect::EffectRecord::Chat::keep: u64 -pub runvil::effect::EffectRecord::Chat::max_tokens: core::option::Option -pub runvil::effect::EffectRecord::Chat::messages: alloc::vec::Vec -pub runvil::effect::EffectRecord::Chat::round: runvil::ids::RoundId -pub runvil::effect::EffectRecord::Chat::temperature: core::option::Option -pub runvil::effect::EffectRecord::Chat::thinking: core::option::Option -pub runvil::effect::EffectRecord::Chat::tools: alloc::vec::Vec -pub runvil::effect::EffectRecord::Render { .. } -pub runvil::effect::EffectRecord::Render::alias: alloc::string::String -pub runvil::effect::EffectRecord::Render::entry: alloc::string::String -pub runvil::effect::EffectRecord::Render::offered: alloc::vec::Vec> -pub runvil::effect::EffectRecord::Timer { .. } -pub runvil::effect::EffectRecord::Timer::seconds: f64 -pub runvil::effect::EffectRecord::ToolCall { .. } -pub runvil::effect::EffectRecord::ToolCall::alias: alloc::string::String -pub runvil::effect::EffectRecord::ToolCall::args: serde_json::value::Value -pub runvil::effect::EffectRecord::ToolCall::origin: runvil::effect::ToolCallOrigin -pub runvil::effect::EffectRecord::ToolCall::tool: runvil::tools::ToolId -pub runvil::effect::EffectRecord::Vfs { .. } -pub runvil::effect::EffectRecord::Vfs::op: runvil::vfs::VfsOp -pub runvil::effect::FailureRecord::kind: alloc::string::String -pub runvil::effect::FailureRecord::message: alloc::string::String pub runvil::effect::IssuedEffect::effect: runvil::effect::Effect pub runvil::effect::IssuedEffect::id: runvil::effect::EffectId pub runvil::effect::IssuedEffect::provenance: runvil::ids::Provenance @@ -933,8 +732,6 @@ pub runvil::effect::PackToolSets::pack: runvil::packs::PackId pub runvil::effect::PackToolSets::static_set: alloc::vec::Vec pub runvil::effect::Round::id: runvil::ids::RoundId pub runvil::effect::Round::origin: runvil::event::ReplyOrigin -pub runvil::effect::ToolAnswerRecord::text: alloc::string::String -pub runvil::effect::ToolAnswerRecord::trust: runvil::tools::OutputTrust pub runvil::effect::ToolCallOrigin::caller: runvil::effect::ToolCaller pub runvil::effect::ToolCallOrigin::entry: core::option::Option pub runvil::effect::ToolCallOrigin::execution: alloc::string::String @@ -942,8 +739,6 @@ pub runvil::effect::ToolCallOrigin::model: core::option::Option @@ -1007,18 +802,6 @@ pub runvil::event::Event::LuaLibraryLoadSucceeded::execution: alloc::string::Str pub runvil::event::Event::LuaLibraryLoadSucceeded::library: alloc::string::String pub runvil::event::Event::LuaLibraryLoadSucceeded::provenance: runvil::ids::Provenance pub runvil::event::Event::LuaLibraryLoadSucceeded::section: alloc::string::String -pub runvil::event::Event::LuaReplyBindingFailed { .. } -pub runvil::event::Event::LuaReplyBindingFailed::execution: alloc::string::String -pub runvil::event::Event::LuaReplyBindingFailed::provenance: runvil::ids::Provenance -pub runvil::event::Event::LuaReplyBindingFailed::section: alloc::string::String -pub runvil::event::Event::LuaReplyBindingStarted { .. } -pub runvil::event::Event::LuaReplyBindingStarted::execution: alloc::string::String -pub runvil::event::Event::LuaReplyBindingStarted::provenance: runvil::ids::Provenance -pub runvil::event::Event::LuaReplyBindingStarted::section: alloc::string::String -pub runvil::event::Event::LuaReplyBindingSucceeded { .. } -pub runvil::event::Event::LuaReplyBindingSucceeded::execution: alloc::string::String -pub runvil::event::Event::LuaReplyBindingSucceeded::provenance: runvil::ids::Provenance -pub runvil::event::Event::LuaReplyBindingSucceeded::section: alloc::string::String pub runvil::event::Event::LuaTeardownStarted { .. } pub runvil::event::Event::LuaTeardownStarted::execution: alloc::string::String pub runvil::event::Event::LuaTeardownStarted::provenance: runvil::ids::Provenance @@ -1027,18 +810,6 @@ pub runvil::event::Event::LuaTeardownSucceeded { .. } pub runvil::event::Event::LuaTeardownSucceeded::execution: alloc::string::String pub runvil::event::Event::LuaTeardownSucceeded::provenance: runvil::ids::Provenance pub runvil::event::Event::LuaTeardownSucceeded::section: alloc::string::String -pub runvil::event::Event::ModelCatalogValidationFailed { .. } -pub runvil::event::Event::ModelCatalogValidationFailed::execution: alloc::string::String -pub runvil::event::Event::ModelCatalogValidationFailed::provenance: runvil::ids::Provenance -pub runvil::event::Event::ModelCatalogValidationFailed::section: alloc::string::String -pub runvil::event::Event::ModelCatalogValidationStarted { .. } -pub runvil::event::Event::ModelCatalogValidationStarted::execution: alloc::string::String -pub runvil::event::Event::ModelCatalogValidationStarted::provenance: runvil::ids::Provenance -pub runvil::event::Event::ModelCatalogValidationStarted::section: alloc::string::String -pub runvil::event::Event::ModelCatalogValidationSucceeded { .. } -pub runvil::event::Event::ModelCatalogValidationSucceeded::execution: alloc::string::String -pub runvil::event::Event::ModelCatalogValidationSucceeded::provenance: runvil::ids::Provenance -pub runvil::event::Event::ModelCatalogValidationSucceeded::section: alloc::string::String pub runvil::event::Event::ModelMetadataDegraded { .. } pub runvil::event::Event::ModelMetadataDegraded::execution: alloc::string::String pub runvil::event::Event::ModelMetadataDegraded::message: alloc::string::String @@ -1119,17 +890,6 @@ pub runvil::event::Event::TaskFailed::execution: alloc::string::String pub runvil::event::Event::TaskFailed::provenance: runvil::ids::Provenance pub runvil::event::Event::TaskFailed::section: alloc::string::String pub runvil::event::Event::TaskFailed::task: runvil::ids::TaskId -pub runvil::event::Event::TaskNote { .. } -pub runvil::event::Event::TaskNote::execution: alloc::string::String -pub runvil::event::Event::TaskNote::provenance: runvil::ids::Provenance -pub runvil::event::Event::TaskNote::section: alloc::string::String -pub runvil::event::Event::TaskNote::task: runvil::ids::TaskId -pub runvil::event::Event::TaskNote::text: alloc::string::String -pub runvil::event::Event::TaskResumed { .. } -pub runvil::event::Event::TaskResumed::execution: alloc::string::String -pub runvil::event::Event::TaskResumed::provenance: runvil::ids::Provenance -pub runvil::event::Event::TaskResumed::section: alloc::string::String -pub runvil::event::Event::TaskResumed::task: runvil::ids::TaskId pub runvil::event::Event::TaskStarted { .. } pub runvil::event::Event::TaskStarted::execution: alloc::string::String pub runvil::event::Event::TaskStarted::index: core::option::Option @@ -1304,11 +1064,14 @@ pub runvil::model::CompletionResult::Text(..) pub runvil::model::CompletionResult::Text::0: alloc::string::String pub runvil::model::CompletionResult::ToolCalls(..) pub runvil::model::CompletionResult::ToolCalls::0: alloc::vec::Vec -pub runvil::model::ModelCatalogError::DuplicateId::name: alloc::string::String -pub runvil::model::ModelCatalogError::DuplicateId::server: alloc::string::String pub runvil::model::ModelInvocation::max_tokens: core::option::Option pub runvil::model::ModelInvocation::temperature: core::option::Option pub runvil::model::ModelInvocation::thinking: core::option::Option +pub runvil::model::ModelRole::Creative +pub runvil::model::ModelRole::Default +pub runvil::model::ModelRole::Fast +pub runvil::model::ModelRole::Small +pub runvil::model::ModelRole::Strong pub runvil::model::Role::Assistant pub runvil::model::Role::System pub runvil::model::Role::Tool @@ -1318,37 +1081,24 @@ pub runvil::model::TemperatureError::OutOfRange::value: f64 pub runvil::model::ThinkingMode::Always pub runvil::model::ThinkingMode::Never pub runvil::model::ThinkingMode::Switchable -pub runvil::packs::PackIdErrorKind::Charset -pub runvil::packs::PackIdErrorKind::Control -pub runvil::packs::PackIdErrorKind::Empty -pub runvil::packs::PackIdErrorKind::SegmentCount +pub runvil::packs::RosterEntry::Unavailable { .. } +pub runvil::packs::RosterEntry::Unavailable::reason: alloc::string::String +pub runvil::packs::RosterEntry::Usable { .. } +pub runvil::packs::RosterEntry::Usable::vendor: alloc::string::String pub runvil::prompt::ArgType::Boolean pub runvil::prompt::ArgType::Integer pub runvil::prompt::ArgType::Number pub runvil::prompt::ArgType::String -pub runvil::prompt::ModelKeyword::Chat -pub runvil::prompt::ModelKeyword::Creative -pub runvil::prompt::ModelKeyword::Fast -pub runvil::prompt::ModelKeyword::Frontier -pub runvil::prompt::ModelKeyword::NoThinking -pub runvil::prompt::ModelKeyword::Small -pub runvil::prompt::ModelKeyword::Thinking pub runvil::tools::OutputTrust::Trusted pub runvil::tools::OutputTrust::Untrusted -pub runvil::tools::ToolCatalogError::DuplicateId::id: runvil::tools::ToolId -pub runvil::tools::ToolCatalogErrorKind::DuplicateId pub runvil::tools::ToolErrorKind::Backend -pub runvil::tools::ToolErrorKind::Cancelled pub runvil::tools::ToolErrorKind::InvalidArguments pub runvil::tools::ToolErrorKind::Other pub runvil::tools::ToolErrorKind::Transport -pub runvil::tools::ToolIdErrorKind::Charset -pub runvil::tools::ToolIdErrorKind::Control -pub runvil::tools::ToolIdErrorKind::Empty -pub runvil::tools::ToolIdErrorKind::SegmentCount pub runvil::tools::ToolInfo::describes_tools: bool pub runvil::tools::ToolInfo::description: alloc::string::String pub runvil::tools::ToolInfo::id: runvil::tools::ToolId +pub runvil::tools::ToolInfo::model_only_call: bool pub runvil::tools::ToolInfo::parameters_schema: serde_json::value::Value pub runvil::tools::ToolInfo::structured_output: bool pub runvil::tools::ToolInfo::survives_stop: bool @@ -1493,27 +1243,23 @@ pub runvil::vfs::VfsOutcome::Unit pub runvil::vfs::Writability::ReadOnly pub runvil::vfs::Writability::ReadWrite pub struct runvil::Run { /* private fields */ } -pub struct runvil::RunContext { /* private fields */ } -pub struct runvil::RunError { /* private fields */ } +pub struct runvil::RunConfig { /* private fields */ } pub struct runvil::RunSeed(/* private fields */) pub struct runvil::cancel::CancelHandle { /* private fields */ } pub struct runvil::cancel::Cancelled { /* private fields */ } pub struct runvil::effect::EffectId(/* private fields */) pub struct runvil::ids::ChainId(/* private fields */) -pub struct runvil::ids::ParseIdError { /* private fields */ } +pub struct runvil::ids::IdError { /* private fields */ } pub struct runvil::ids::RoundId(/* private fields */) pub struct runvil::ids::TaskId(/* private fields */) pub struct runvil::model::ArgumentNames<'a> { /* private fields */ } pub struct runvil::model::ModelBinding { /* private fields */ } -pub struct runvil::model::ModelBindings { /* private fields */ } pub struct runvil::model::ParseRoleError { /* private fields */ } pub struct runvil::model::Temperature(/* private fields */) pub struct runvil::packs::GlobalName { /* private fields */ } pub struct runvil::packs::PackRef { /* private fields */ } pub struct runvil::packs::Prelude { /* private fields */ } pub struct runvil::prompt::ArgsDeclIter<'a> { /* private fields */ } -pub struct runvil::prompt::ModelRolesIter<'a> { /* private fields */ } -pub struct runvil::replay::Flags(/* private fields */) pub struct runvil::timestamp::Timestamp(/* private fields */) pub struct runvil::vfs::Access { /* private fields */ } pub struct runvil::vfs::AcquireContext { /* private fields */ } diff --git a/crates/runvil/src/lib.rs b/crates/runvil/src/lib.rs index 19acb1fa0..8714b0960 100644 --- a/crates/runvil/src/lib.rs +++ b/crates/runvil/src/lib.rs @@ -4,26 +4,22 @@ //! A [`Prompt`] is a parsed prompt file. A [`Run`] executes one prompt. //! [`Run::step`] returns [`Step::Pending`] with the effects the run waits on. //! The caller performs each effect and hands its answer to [`Run::resume`]. -//! The run ends at [`Step::Done`] with its [`RunResult`]. -//! Before the run starts, the caller puts the tools a run may offer on its -//! [`RunContext`], and [`RunContext::prepare`] binds the prompt's model roles. +//! The run ends at [`Step::Done`] with its [`RunOutcome`]. +//! Before the run starts, the caller puts on its [`RunConfig`] the tools +//! a run may offer, the Packs it holds, and the role map that fills the +//! five model roles. The run's first step checks the config against what +//! the prompt requires, and a config that falls short ends the run there +//! with a [`RequirementsUnmet`](RunErrorKind::RequirementsUnmet) failure +//! that names every gap. //! //! The crate's `greeter` example is that whole loop in one program. -pub use runvil_engine::MissingService; -pub use runvil_engine::RequirementCheck; -pub use runvil_engine::Requirements; pub use runvil_engine::Run; -pub use runvil_engine::RunContext; -pub use runvil_engine::RunError; +pub use runvil_engine::RunConfig; pub use runvil_engine::RunErrorKind; pub use runvil_engine::RunLimits; -pub use runvil_engine::RunResult; -pub use runvil_engine::SourceLocation; +pub use runvil_engine::RunOutcome; pub use runvil_engine::Step; -pub use runvil_engine::UnavailablePack; -pub use runvil_engine::UnmetRequirement; -pub use runvil_engine::UsablePack; pub use runvil_parser::ParseError; pub use runvil_parser::ParseErrorKind; pub use runvil_parser::Prompt; @@ -33,17 +29,12 @@ pub mod effect { //! The outside work a run asks for, the answers it takes back, and the //! records a run log stores for both. - pub use runvil_engine::AnswerRecord; - pub use runvil_engine::ChatAnswerRecord; pub use runvil_engine::Effect; pub use runvil_engine::EffectAnswer; pub use runvil_engine::EffectId; - pub use runvil_engine::EffectRecord; - pub use runvil_engine::FailureRecord; pub use runvil_engine::IssuedEffect; pub use runvil_engine::PackToolSets; pub use runvil_engine::Round; - pub use runvil_engine::ToolAnswerRecord; pub use runvil_engine::ToolCallOrigin; pub use runvil_engine::ToolCaller; } @@ -51,28 +42,27 @@ pub mod effect { pub mod event { //! The events a run reports for the caller to log. - pub use runvil_types::emitter::DebugMode; pub use runvil_types::event::Event; pub use runvil_types::event::ReplyOrigin; } pub mod ids { - //! Task, chain, and round ids, and the provenance that orders a run's - //! records. + //! Task, chain, and round ids, the provenance that orders a run's + //! records, and the error an invalid id, or a catalog that repeats one, + //! reports. pub use runvil_types::ids::AbandonReason; pub use runvil_types::ids::ChainId; - pub use runvil_types::ids::ParseIdError; pub use runvil_types::ids::Provenance; pub use runvil_types::ids::RoundId; pub use runvil_types::ids::TaskId; + pub use runvil_types::names::IdError; } pub mod model { //! Model descriptions and catalogs, the models a prompt's roles are //! bound to, and the chat vocabulary of a model round. - pub use runvil_engine::ModelBindings; pub use runvil_model_client::client::ArgumentNames; pub use runvil_model_client::client::Completion; pub use runvil_model_client::client::CompletionResult; @@ -91,10 +81,9 @@ pub mod model { pub use runvil_model_client::model::Temperature; pub use runvil_model_client::model::TemperatureError; pub use runvil_types::models::ModelCatalog; - pub use runvil_types::models::ModelCatalogError; pub use runvil_types::models::ModelId; - pub use runvil_types::models::ModelIdError; pub use runvil_types::models::ModelInfo; + pub use runvil_types::models::ModelRole; pub use runvil_types::models::ThinkingMode; } @@ -104,27 +93,21 @@ pub mod tools { pub use runvil_types::tools::OutputTrust; pub use runvil_types::tools::ToolCatalog; - pub use runvil_types::tools::ToolCatalogError; - pub use runvil_types::tools::ToolCatalogErrorKind; pub use runvil_types::tools::ToolError; pub use runvil_types::tools::ToolErrorKind; pub use runvil_types::tools::ToolId; - pub use runvil_types::tools::ToolIdError; - pub use runvil_types::tools::ToolIdErrorKind; pub use runvil_types::tools::ToolInfo; pub use runvil_types::tools::ToolOutput; } pub mod packs { - //! Pack ids, the `uses:` entries that declare Packs, the Lua - //! preludes Packs add to a run, and the global names those preludes - //! may define. + //! Pack ids, the `uses:` entries that declare Packs, how each Pack + //! stands in a run's roster, the Lua preludes Packs add to a run, and + //! the global names those preludes may define. + pub use runvil_lua::RosterEntry; pub use runvil_types::names::GlobalName; - pub use runvil_types::names::GlobalNameError; pub use runvil_types::packs::PackId; - pub use runvil_types::packs::PackIdError; - pub use runvil_types::packs::PackIdErrorKind; pub use runvil_types::packs::PackRef; pub use runvil_types::packs::Prelude; } @@ -139,10 +122,7 @@ pub mod prompt { pub use runvil_parser::ArgsDeclIter; pub use runvil_parser::FileDecl; pub use runvil_parser::Frontmatter; - pub use runvil_parser::ModelKeyword; - pub use runvil_parser::ModelRole; - pub use runvil_parser::ModelRoles; - pub use runvil_parser::ModelRolesIter; + pub use runvil_parser::RoleRequirements; } pub mod vfs { @@ -189,7 +169,7 @@ pub mod cancel { } pub mod timestamp { - //! The start time a run's context carries, which a prompt reads as + //! The start time a run's config carries, which a prompt reads as //! `sys.when`. pub use runvil_types::timestamp::Timestamp; @@ -205,9 +185,3 @@ pub mod metrics { pub use runvil_types::metrics::Usage; pub use runvil_types::metrics::VllmMetrics; } - -pub mod replay { - //! The behavior flags a run's record keeps and hands back bit for bit. - - pub use runvil_types::replay::Flags; -} diff --git a/crates/runvil/tests/suite/cancel.rs b/crates/runvil/tests/suite/cancel.rs index cbcc0b0b5..5563bc06a 100644 --- a/crates/runvil/tests/suite/cancel.rs +++ b/crates/runvil/tests/suite/cancel.rs @@ -7,7 +7,7 @@ use std::sync::Arc; use std::task::{Context, Poll, Wake, Waker}; use std::thread::{self, Thread}; -use runvil::RunResult; +use runvil::RunOutcome; use runvil::cancel::CancelHandle; use runvil::effect::Effect; @@ -42,7 +42,7 @@ fn one_parent_cancel_ends_every_run_holding_one_of_its_children() -> TestResult let parent = parent.clone(); let mut driver = Scenario::default() .hold(|effect| matches!(effect, Effect::Vfs { .. })) - .context(move |ctx| ctx.with_cancel(parent.child())) + .config(move |ctx| ctx.with_cancel(parent.child())) .start(&source)?; driver.pump_until_held()?; drivers.push(driver); @@ -50,23 +50,21 @@ fn one_parent_cancel_ends_every_run_holding_one_of_its_children() -> TestResult parent.cancel(); - let mut results = Vec::new(); + let mut outcomes = Vec::new(); for driver in drivers { - results.push(driver.finish()?.result); + outcomes.push(driver.finish()?.outcome); } - assert!( - results.len() == 2 - && results - .iter() - .all(|result| matches!(result, RunResult::Cancelled)) - ); + assert_eq!(outcomes, [RunOutcome::Cancelled, RunOutcome::Cancelled]); Ok(()) } #[test] fn a_cancelled_future_wakes_when_another_thread_cancels_the_run() -> TestResult { - let driver = Scenario::default().start(¬e("'hello'"))?; - let handle = driver.run.cancel_handle(); + let handle = CancelHandle::new(); + let handed = handle.clone(); + let driver = Scenario::default() + .config(move |ctx| ctx.with_cancel(handed.clone())) + .start(¬e("'hello'"))?; let waiting = handle.cancelled(); let remote = handle.clone(); @@ -77,5 +75,10 @@ fn a_cancelled_future_wakes_when_another_thread_cancels_the_run() -> TestResult .join() .map_err(|_| "the cancelling thread panicked")?; assert!(handle.is_cancelled()); + assert_eq!( + driver.finish()?.outcome, + RunOutcome::Cancelled, + "the run reads the flag the caller handed its config" + ); Ok(()) } diff --git a/crates/runvil/tests/suite/effect.rs b/crates/runvil/tests/suite/effect.rs index 01a77724e..abaa8440e 100644 --- a/crates/runvil/tests/suite/effect.rs +++ b/crates/runvil/tests/suite/effect.rs @@ -1,6 +1,6 @@ //! Answering every kind of effect, and logging each with its answer. -use runvil::effect::{AnswerRecord, EffectRecord, ToolCallOrigin, ToolCaller}; +use runvil::effect::{ToolCallOrigin, ToolCaller}; use serde_json::{Value, json}; use super::support::{Scenario, TestResult, prompt, shout, text}; @@ -15,7 +15,7 @@ const GREET: &str = " "; fn greeter() -> String { - let ask = "models.use('writer')\nreturn models.infer('hello')"; + let ask = "return models.infer('hello')"; let shout = "return tools.call('example/text/shout', { text = 'hi there' })"; let sections = [ ("Greet", GREET), @@ -23,7 +23,7 @@ fn greeter() -> String { ("Shout", shout), ("Quick", "return 'quick'"), ]; - prompt("models: { writer: {} }", §ions) + prompt("", §ions) } fn scenario() -> Scenario { @@ -59,11 +59,9 @@ fn a_tool_calls_origin_is_named_at_runvil_effect() -> TestResult { let origin = trace .effects .iter() - .find_map(|logged| match &logged.effect { - EffectRecord::ToolCall { origin, .. } => Some(origin.clone()), - _ => None, - }) + .find_map(|logged| logged.effect_of("ToolCall")) .ok_or("the greeter calls its shout tool before it ends")?; + let origin: ToolCallOrigin = serde_json::from_value(origin["origin"].clone())?; let expected = ToolCallOrigin::new("greeter", "Shout", ToolCaller::Script).with_entry("0.1.0"); assert_eq!(origin, expected); Ok(()) @@ -91,9 +89,15 @@ fn each_effect_and_its_answer_log_as_one_json_line_that_parses_back() -> TestRes .map(|line| serde_json::from_str(line)) .collect::>()?; assert_eq!(lines.len(), trace.effects.len()); - for line in &lines { - let _: EffectRecord = serde_json::from_value(line["effect"].clone())?; - let _: AnswerRecord = serde_json::from_value(line["answer"].clone())?; + for (line, logged) in lines.iter().zip(&trace.effects) { + assert_eq!( + line["effect"], logged.effect, + "the effect reads back exactly" + ); + assert_eq!( + line["answer"], logged.answer, + "the answer reads back exactly" + ); } let write = lines diff --git a/crates/runvil/tests/suite/event.rs b/crates/runvil/tests/suite/event.rs index dab3792e6..5210dbd05 100644 --- a/crates/runvil/tests/suite/event.rs +++ b/crates/runvil/tests/suite/event.rs @@ -2,21 +2,19 @@ use std::collections::HashSet; -use runvil::UsablePack; -use runvil::event::{DebugMode, Event}; -use runvil::packs::PackId; +use runvil::event::Event; +use runvil::packs::{PackId, RosterEntry}; use serde_json::Value; use super::support::{Scenario, TestResult, note, prompt, shout, text}; const SHOUT: &str = " store.write('note.md', 'hello ' .. args) - models.use('writer') return tools.call('example/text/shout', { text = models.infer(store.read('note.md')) }) "; fn shouting() -> (String, Scenario) { - let source = prompt("models: { writer: {} }", &[("Greet", SHOUT)]); + let source = prompt("", &[("Greet", SHOUT)]); ( source, Scenario::tool(shout(), [text("hi there")]).args("world"), @@ -40,7 +38,7 @@ fn every_event_round_trips_as_a_json_line_and_no_two_share_a_provenance() -> Tes assert_eq!(keys.len(), trace.events.len()); let quiet = Scenario::default() .args("world") - .context(|ctx| ctx.with_provenance_start(0)) + .config(|ctx| ctx.with_provenance_start(0)) .run(&source)?; assert_eq!(quiet.text()?, "hello world"); Ok(()) @@ -50,8 +48,11 @@ fn every_event_round_trips_as_a_json_line_and_no_two_share_a_provenance() -> Tes fn content_events_pair_each_tool_result_with_its_caller_into_a_transcript() -> TestResult { let (source, scenario) = shouting(); let example = PackId::parse("example")?; + let usable = RosterEntry::Usable { + vendor: "acme".to_owned(), + }; let trace = scenario - .context(move |ctx| ctx.with_packs(vec![UsablePack::new(example.clone(), "acme")], vec![])) + .config(move |ctx| ctx.with_packs([(example.clone(), usable.clone())])) .run(&source)?; assert_eq!(trace.text()?, "HI THERE"); @@ -104,7 +105,7 @@ fn content_events_pair_each_tool_result_with_its_caller_into_a_transcript() -> T fn debug_mode_alone_adds_one_request_and_one_response_per_model_round() -> TestResult { let (source, scenario) = shouting(); let captured = scenario - .context(|ctx| ctx.with_report_debug(DebugMode::On)) + .config(|ctx| ctx.with_report_debug(true)) .run(&source)?; let (mut requests, mut responses) = (Vec::new(), Vec::new()); diff --git a/crates/runvil/tests/suite/greeter.rs b/crates/runvil/tests/suite/greeter.rs index c7648e241..90bd69daf 100644 --- a/crates/runvil/tests/suite/greeter.rs +++ b/crates/runvil/tests/suite/greeter.rs @@ -1,8 +1,9 @@ //! The greeter, one tour at a time, up to the complete program in //! `examples/greeter.rs`. +use runvil::cancel::CancelHandle; use runvil::effect::{Effect, EffectAnswer}; -use runvil::{Prompt, RunResult}; +use runvil::{Prompt, RunOutcome}; use super::support::{Scenario, TestResult, note, prompt, shout, text}; @@ -18,7 +19,6 @@ use example::{GREETER, answer}; const REPLY: &str = " store.write('note.md', 'hello') - models.use('writer') return models.infer(store.read('note.md')) "; @@ -46,8 +46,9 @@ fn the_example_answers_a_timer_as_fired_at_once() { } #[test] -fn prepare_fills_the_writer_role_and_the_canned_reply_becomes_the_result() -> TestResult { - let source = prompt("models: { writer: {} }", &[("Greet", REPLY)]); +fn a_prompt_declaring_no_models_runs_on_the_default_model_and_the_canned_reply_becomes_the_result() +-> TestResult { + let source = prompt("", &[("Greet", REPLY)]); let trace = Scenario::replies([text("hi there")]).run(&source)?; assert_eq!(trace.text()?, "hi there"); Ok(()) @@ -62,18 +63,23 @@ fn trusted_tool_output_reaches_the_calling_lua_unchanged() -> TestResult { #[test] fn a_cancel_from_another_thread_ends_the_run_cancelled_once_held_effects_drop() -> TestResult { + let cancel = CancelHandle::new(); + let handed = cancel.clone(); let mut driver = Scenario::tool(shout(), []) .hold(|effect| matches!(effect, Effect::Chat { .. })) + .config(move |ctx| ctx.with_cancel(handed.clone())) .start(GREETER)?; driver.pump_until_held()?; - let handle = driver.run.cancel_handle(); - std::thread::spawn(move || handle.cancel()) + std::thread::spawn(move || cancel.cancel()) .join() .map_err(|_| "the cancelling thread panicked")?; - assert!(driver.pump()? == 0 && driver.run.decided()); - assert!(matches!(driver.finish()?.result, RunResult::Cancelled)); + assert!( + driver.pump()? == 0 && driver.held() == 0, + "the step that decides the run withdraws the held chat effect" + ); + assert_eq!(driver.finish()?.outcome, RunOutcome::Cancelled); Ok(()) } diff --git a/crates/runvil/tests/suite/ids.rs b/crates/runvil/tests/suite/ids.rs index 3c82f57b4..0f7f64143 100644 --- a/crates/runvil/tests/suite/ids.rs +++ b/crates/runvil/tests/suite/ids.rs @@ -15,7 +15,6 @@ const GREET: &str = " "; const REPLY: &str = " - models.use('writer') return models.infer(args) "; @@ -23,10 +22,7 @@ type Groups = BTreeMap>; /// Returns the started tasks beside the run's groups. fn run_greeter() -> TestResult<(Vec, Groups)> { - let source = prompt( - "models: { writer: {} }", - &[("Greet", GREET), ("Reply", REPLY)], - ); + let source = prompt("", &[("Greet", GREET), ("Reply", REPLY)]); let trace = Scenario::replies([text("hi there"), text("hi there")]).run(&source)?; assert_eq!(trace.text()?, "hi there hi there"); let mut groups = Groups::new(); @@ -74,6 +70,9 @@ fn parsed_task_ids_compare_as_values_and_a_damaged_id_reports_its_whole_text() - .parse::() .err() .ok_or("0..1 is not a task id")?; - assert_eq!(error.input(), "0..1"); + assert_eq!( + error.to_string(), + "invalid chain id `0..1`: required a dot-separated path of decimal components" + ); Ok(()) } diff --git a/crates/runvil/tests/suite/main.rs b/crates/runvil/tests/suite/main.rs index e8dbbb426..c6d01644a 100644 --- a/crates/runvil/tests/suite/main.rs +++ b/crates/runvil/tests/suite/main.rs @@ -10,7 +10,6 @@ mod packs; mod parsing; mod prepare; mod prompt; -mod replay; mod shipped; mod support; mod timestamp; diff --git a/crates/runvil/tests/suite/model.rs b/crates/runvil/tests/suite/model.rs index b40a692ee..847de95e8 100644 --- a/crates/runvil/tests/suite/model.rs +++ b/crates/runvil/tests/suite/model.rs @@ -1,22 +1,21 @@ //! Describing models and answering their rounds. -use runvil::model::{CompletionErrorKind, ModelCatalog, ModelCatalogError, ModelId, ThinkingMode}; -use runvil::{RunErrorKind, RunResult}; +use runvil::model::{CompletionErrorKind, ModelCatalog, ModelId, ThinkingMode}; +use runvil::{RunErrorKind, RunOutcome}; use super::support::{Reply, Scenario, TestResult, fail, model, prompt}; fn asking() -> String { let lua = " store.write('note.md', 'hello') - models.use('writer') return models.infer(store.read('note.md')) "; - prompt("models: { writer: {}, checker: {} }", &[("Greet", lua)]) + prompt("", &[("Greet", lua)]) } -fn run_asking(reply: Reply) -> TestResult { +fn run_asking(reply: Reply) -> TestResult { let scenario = Scenario::replies([reply]).model(model("fast", 32_000, ThinkingMode::Never)?); - Ok(scenario.run(&asking())?.result) + Ok(scenario.run(&asking())?.outcome) } #[test] @@ -32,20 +31,21 @@ fn a_catalog_finds_a_model_by_id_and_refuses_a_repeated_id_as_a_whole() -> TestR assert_eq!(found.context().get(), 200_000); assert_eq!(found.thinking(), ThinkingMode::Always); - let Err(ModelCatalogError::DuplicateId { server, name, .. }) = - ModelCatalog::new([fast.clone(), fast]) - else { - return Err("a repeated id is refused with DuplicateId".into()); - }; - assert_eq!((server.as_str(), name.as_str()), ("gateway", "fast")); + let error = ModelCatalog::new([fast.clone(), fast]) + .err() + .ok_or("a repeated id is refused")?; + assert_eq!( + error.to_string(), + "duplicate model identity in catalog: gateway/fast" + ); Ok(()) } #[test] fn a_failed_model_round_fails_the_run_as_a_completion_failure() -> TestResult { - let RunResult::Failure(error) = run_asking(fail(CompletionErrorKind::Overloaded))? else { + let RunOutcome::Failed { kind, .. } = run_asking(fail(CompletionErrorKind::Overloaded))? else { return Err("a failed round fails the run".into()); }; - assert_eq!(error.kind(), RunErrorKind::Completion); + assert_eq!(kind, RunErrorKind::Completion); Ok(()) } diff --git a/crates/runvil/tests/suite/packs.rs b/crates/runvil/tests/suite/packs.rs index 426c1b376..bd96a19a2 100644 --- a/crates/runvil/tests/suite/packs.rs +++ b/crates/runvil/tests/suite/packs.rs @@ -1,21 +1,32 @@ //! Handing Packs to a run. -use runvil::packs::PackId; -use runvil::{UnavailablePack, UsablePack}; +use runvil::packs::{PackId, RosterEntry}; -use super::support::{Scenario, TestResult, greeter}; +use super::prepare::refusal; +use super::support::{Scenario, TestResult, greeter, prompt}; #[test] -fn two_runs_whose_contexts_carry_different_rosters_each_see_their_own() -> TestResult { +fn two_runs_whose_configs_carry_different_rosters_each_see_their_own() -> TestResult { let source = greeter("return tostring(tools.packs.get('kit') ~= nil)"); let kit = PackId::parse("kit")?; let usable = { let kit = kit.clone(); - Scenario::default() - .context(move |ctx| ctx.with_packs(vec![UsablePack::new(kit.clone(), "acme")], vec![])) + Scenario::default().config(move |ctx| { + ctx.with_packs([( + kit.clone(), + RosterEntry::Usable { + vendor: "acme".to_owned(), + }, + )]) + }) }; - let unavailable = Scenario::default().context(move |ctx| { - ctx.with_packs(vec![], vec![UnavailablePack::new(kit.clone(), "down")]) + let unavailable = Scenario::default().config(move |ctx| { + ctx.with_packs([( + kit.clone(), + RosterEntry::Unavailable { + reason: "down".to_owned(), + }, + )]) }); for (scenario, expected) in [(usable, "true"), (unavailable, "false")] { @@ -23,3 +34,24 @@ fn two_runs_whose_contexts_carry_different_rosters_each_see_their_own() -> TestR } Ok(()) } + +#[test] +fn a_declared_pack_the_roster_lacks_or_cannot_use_refuses_the_run() -> TestResult { + let source = prompt("uses: [kit, web]", &[("Only", "return 'done'")]); + let kit = PackId::parse("kit")?; + let scenario = Scenario::default().config(move |ctx| { + ctx.with_packs([( + kit.clone(), + RosterEntry::Unavailable { + reason: "the kit server is down".to_owned(), + }, + )]) + }); + assert_eq!( + refusal(scenario, &source)?, + "the environment cannot satisfy this prompt:\n\ + - missing required Pack: web\n\ + - kit is unavailable: the kit server is down" + ); + Ok(()) +} diff --git a/crates/runvil/tests/suite/prepare.rs b/crates/runvil/tests/suite/prepare.rs index 390e63118..565d9ebb5 100644 --- a/crates/runvil/tests/suite/prepare.rs +++ b/crates/runvil/tests/suite/prepare.rs @@ -1,31 +1,43 @@ -//! The prepare pass: real-file claims, the caller's tool catalog, and model -//! roles. The store-mount isolation case and the prepare-run refusals -//! drive the run fixture in `runvil-engine`'s test support, so they -//! sit in that crate's own suite. +//! A run's start: real-file claims, the caller's tool catalog, and model +//! roles, whose requirements the run's first step checks. The store-mount +//! isolation case and the start-check refusals that reach engine-only +//! items sit in `runvil-engine`'s own suite. -use runvil::model::ThinkingMode; -use runvil::tools::{ToolCatalog, ToolInfo}; +use runvil::event::Event; +use runvil::model::{ModelRole, ThinkingMode}; +use runvil::tools::{ToolCatalog, ToolInfo, ToolOutput}; use runvil::vfs::{MemoryBackend, Origin, RealBackend, VfsError, VfsRef}; -use runvil::{RequirementCheck, Requirements, UnmetRequirement}; +use runvil::{RunErrorKind, RunOutcome}; -use super::support::{Prepared, Scenario, TempRoot, TestResult, greeter, model, prompt, tool_info}; +use super::support::{Scenario, TempRoot, TestResult, Tool, Trace, model, prompt, text, tool_info}; fn declaring(front: &str) -> String { prompt(front, &[("Only", "return 'done'")]) } -fn report(front: &str, context: u32, thinking: ThinkingMode) -> TestResult { - let prepared = Scenario::default() - .model(model("current", context, thinking)?) - .prepare(&declaring(front))?; - Ok(prepared.requirements) +/// The refusal notice of `scenario`'s run of `source`, after checking that +/// its first step failed it as `RequirementsUnmet`, reporting nothing +/// after the parse events and issuing no effect. +pub(super) fn refusal(scenario: Scenario, source: &str) -> TestResult { + let trace = scenario.run(source)?; + let RunOutcome::Failed { kind, message } = &trace.outcome else { + return Err(format!("expected a refusal, got {:?}", trace.outcome).into()); + }; + assert_eq!(*kind, RunErrorKind::RequirementsUnmet, "{message}"); + assert!( + matches!(trace.events.last(), Some(Event::ParseSucceeded { .. })), + "a refused run reports no run events: {:?}", + trace.events + ); + assert!(trace.effects.is_empty(), "a refused run issues nothing"); + Ok(message.clone()) } -fn only_unmet(requirements: &Requirements) -> TestResult<&UnmetRequirement> { - match requirements.unmet_models.as_slice() { - [unmet] => Ok(unmet), - unmet => Err(format!("exactly one requirement is unmet: {unmet:?}").into()), - } +/// The refusal notice of a run of `declaring(front)` under one model of +/// the given context window and thinking capability. +fn refusal_under(front: &str, context: u32, thinking: ThinkingMode) -> TestResult { + let scenario = Scenario::default().model(model("current", context, thinking)?); + refusal(scenario, &declaring(front)) } #[test] @@ -36,12 +48,10 @@ fn two_runs_writing_the_same_real_file_through_the_shared_base_conflict() -> Tes .mount("/", RealBackend::rooted(temp.path())?) .store("/my/store", MemoryBackend::new()) .build(); - let scenario = Scenario::default().context(move |ctx| ctx.with_vfs(vfs.clone())); - let source = greeter("return 'done'"); - let (run_a, run_b) = (scenario.prepare(&source)?, scenario.prepare(&source)?); - let access_a = run_a.ctx.vfs().acquire(Origin::new("run-a"))?; + let (run_a, run_b) = (vfs.clone(), vfs); + let access_a = run_a.acquire(Origin::new("run-a"))?; access_a.write("/shared.txt", b"from a")?; - let access_b = run_b.ctx.vfs().acquire(Origin::new("run-b"))?; + let access_b = run_b.acquire(Origin::new("run-b"))?; let error = access_b .write("/shared.txt", b"from b") .expect_err("run b conflicts with run a's live claim"); @@ -53,100 +63,151 @@ fn two_runs_writing_the_same_real_file_through_the_shared_base_conflict() -> Tes Ok(()) } -const ANALYST: &str = "models: { analyst: { keywords: [frontier, thinking], min_context: 200000, description: Deep analysis } }"; +const STRONG: &str = "models: { strong: { thinking: true, min_context: 200000 } }"; -#[test] -fn every_declared_role_resolves_to_the_current_model() -> TestResult { - // Soft keywords document author intent, and are never checked. - let source = declaring( - "models: { scout: { keywords: [frontier, fast] }, sprinter: { keywords: [small, creative, chat] } }", +/// The five roles a prompt can pick with `models.use`. +const ROLES: [&str; 5] = ["default", "strong", "small", "fast", "creative"]; + +/// A prompt that asks each of the five roles one question and returns the +/// answers joined by commas. +fn asking_every_role(front: &str) -> String { + let lua = format!( + "local out = {{}}\nfor _, role in ipairs({{'{}'}}) do\nmodels.use(role)\nout[#out + 1] = models.infer(role)\nend\nreturn table.concat(out, ',')", + ROLES.join("','") ); + prompt(front, &[("Only", &lua)]) +} + +/// The models the replies of `trace` name, in order. +fn replying_models(trace: &Trace) -> Vec<&str> { + trace + .events + .iter() + .filter_map(|event| match event { + Event::AssistantReply { model, .. } => Some(model.as_str()), + _ => None, + }) + .collect() +} + +#[test] +fn one_model_fills_every_role_and_an_empty_declaration_adds_no_requirement() -> TestResult { + // A role declared with no requirement checks nothing. + let source = asking_every_role("models: { fast: {}, creative: {} }"); let model = model("current", 32_000, ThinkingMode::Never)?; - let Prepared { - ctx, requirements, .. - } = Scenario::default().model(model.clone()).prepare(&source)?; - assert!(requirements.is_satisfied()); - let bindings = ctx.model_bindings(); - assert_eq!(bindings.len(), 2); - assert_eq!(bindings.role_id("scout"), Some(model.id())); - assert_eq!(bindings.resolve("scout"), Some(&model)); - assert_eq!(bindings.resolve("sprinter"), Some(&model)); - assert_eq!(bindings.model(model.id()), Some(&model)); - assert!(bindings.resolve("undeclared").is_none()); + let replies = ["a", "b", "c", "d", "e"].map(text); + let trace = Scenario::replies(replies).model(model).run(&source)?; + assert_eq!(trace.text()?, "a,b,c,d,e"); + assert_eq!(replying_models(&trace), ["current"; 5]); + Ok(()) +} + +#[test] +fn a_strong_entry_fills_strong_and_the_default_fills_every_other_role() -> TestResult { + let source = asking_every_role(""); + let (default, strong) = ( + model("default-model", 32_000, ThinkingMode::Switchable)?, + model("strong-model", 200_000, ThinkingMode::Switchable)?, + ); + let replies = ["a", "b", "c", "d", "e"].map(text); + let trace = Scenario::replies(replies) + .roles([(ModelRole::Default, default), (ModelRole::Strong, strong)]) + .run(&source)?; + assert_eq!(trace.text()?, "a,b,c,d,e"); + assert_eq!( + replying_models(&trace), + [ + "default-model", + "strong-model", + "default-model", + "default-model", + "default-model" + ], + "the rounds ran in the order of {ROLES:?}" + ); Ok(()) } #[test] -fn a_context_minimum_above_the_current_models_is_reported() -> TestResult { - let requirements = report(ANALYST, 32_000, ThinkingMode::Always)?; - assert!(!requirements.is_satisfied()); - assert_eq!(requirements.missing_packs, []); - let unmet = only_unmet(&requirements)?; - assert_eq!(unmet.role, "analyst"); - assert_eq!(unmet.check, RequirementCheck::ContextMinimum); - assert_eq!(unmet.required, "200000"); - assert_eq!(unmet.actual, "32000"); +fn a_role_map_with_models_but_no_default_is_refused_naming_the_default_role() -> TestResult { + let strong = model("strong-model", 200_000, ThinkingMode::Switchable)?; + let scenario = Scenario::default().roles([(ModelRole::Strong, strong)]); + assert_eq!( + refusal(scenario, &declaring(""))?, + "the environment cannot satisfy this prompt:\n\ + - role 'default': requires a model; the caller's role map has no 'default' entry" + ); Ok(()) } #[test] -fn a_hard_keyword_the_current_model_fails_is_reported() -> TestResult { - // The context minimum is met, so only the keyword check fires. - let requirements = report(ANALYST, 200_000, ThinkingMode::Never)?; - let unmet = only_unmet(&requirements)?; - assert_eq!(unmet.role, "analyst"); - assert_eq!(unmet.check, RequirementCheck::HardKeyword); - assert_eq!(unmet.required, "thinking"); - assert_eq!(unmet.actual, "Never"); - - let triage = "models: { triage: { keywords: [no-thinking] } }"; - let requirements = report(triage, 32_000, ThinkingMode::Always)?; - let unmet = only_unmet(&requirements)?; - assert_eq!(unmet.role, "triage"); - assert_eq!(unmet.check, RequirementCheck::HardKeyword); - assert_eq!(unmet.required, "no-thinking"); - assert_eq!(unmet.actual, "Always"); +fn a_context_minimum_above_the_filling_models_is_refused() -> TestResult { + // The model can think, so only the context check fails. + assert_eq!( + refusal_under(STRONG, 32_000, ThinkingMode::Always)?, + "the environment cannot satisfy this prompt:\n\ + - role 'strong': requires a context of at least 200000 tokens; \ + the model filling it provides 32000" + ); Ok(()) } -fn prepared_over(info: &ToolInfo) -> TestResult { +#[test] +fn a_thinking_requirement_the_filling_model_fails_is_refused() -> TestResult { + // The context minimum is met, so only the thinking check fires. + assert_eq!( + refusal_under(STRONG, 200_000, ThinkingMode::Never)?, + "the environment cannot satisfy this prompt:\n\ + - role 'strong': requires 'thinking: true'; \ + the thinking capability of the model filling it is Never" + ); + let small = "models: { small: { thinking: false } }"; + assert_eq!( + refusal_under(small, 32_000, ThinkingMode::Always)?, + "the environment cannot satisfy this prompt:\n\ + - role 'small': requires 'thinking: false'; \ + the thinking capability of the model filling it is Always" + ); + Ok(()) +} + +/// `scenario` with its catalog replaced by one that holds `info` alone. +fn over_catalog(scenario: Scenario, info: &ToolInfo) -> TestResult { let catalog = ToolCatalog::new([info.clone()])?; - Scenario::default() - .context(move |ctx| ctx.with_tools(catalog.clone())) - .prepare(&declaring("uses: [web]")) + Ok(scenario.config(move |ctx| ctx.with_tools(catalog.clone()))) } #[test] -fn a_prepared_context_keeps_the_caller_supplied_catalog() -> TestResult { +fn a_run_calls_its_tools_as_the_caller_supplied_catalog_describes_them() -> TestResult { + // Only the caller's entry marks the output structured, so the script + // reads a table field only when the run uses that entry. let info = tool_info( "web/fetch", serde_json::json!({"type": "object", "properties": {"url": {"type": "string"}}}), )? .structured(true); - let Prepared { - ctx, requirements, .. - } = prepared_over(&info)?; - assert!( - requirements.is_satisfied(), - "a catalog reports nothing: {requirements:?}" - ); - assert_eq!(ctx.tools().tools(), [info]); + let fetch = Tool { + id: "web/fetch", + handler: Box::new(|_| Ok(ToolOutput::trusted(r#"{"title":"Ada"}"#))), + }; + let scenario = over_catalog(Scenario::tool(fetch, []), &info)?; + let lua = "return tools.call('web/fetch', { url = 'https://example.com' }).title"; + let source = prompt("uses: [web]", &[("Only", lua)]); + assert_eq!(scenario.run(&source)?.text()?, "Ada"); Ok(()) } #[test] -fn prepare_reports_nothing_for_a_declared_pack_the_catalog_lacks() -> TestResult { +fn a_run_given_no_roster_checks_no_declared_pack_against_the_catalog() -> TestResult { let other = tool_info( "other/search", serde_json::json!({"type": "object", "properties": {}}), )?; - let Prepared { - ctx, requirements, .. - } = prepared_over(&other)?; - assert!( - requirements.is_satisfied(), - "the Engine's prepare checks model roles only: {requirements:?}" + let scenario = over_catalog(Scenario::default(), &other)?; + assert_eq!( + scenario.run(&declaring("uses: [web]"))?.text()?, + "done", + "without a roster, the run checks model roles and reserved ids only" ); - assert_eq!(ctx.tools().tools(), [other]); Ok(()) } diff --git a/crates/runvil/tests/suite/prompt.rs b/crates/runvil/tests/suite/prompt.rs index 5855798ca..3626a9d74 100644 --- a/crates/runvil/tests/suite/prompt.rs +++ b/crates/runvil/tests/suite/prompt.rs @@ -1,17 +1,17 @@ //! Reading what a prompt declares before it runs. -use runvil::model::ThinkingMode; -use runvil::prompt::ModelKeyword; -use runvil::{ParseErrorKind, Prompt, RequirementCheck}; +use runvil::model::{ModelRole, ThinkingMode}; +use runvil::{ParseErrorKind, Prompt}; -use super::support::{Prepared, Scenario, TestResult, model, prompt}; +use super::prepare::refusal; +use super::support::{Built, Scenario, TestResult, model, prompt}; const CONTRACT: &str = " input: { path: names.md, description: The names to greet } output: { path: note.md, description: The note the greeter leaves } uses: [text] args: { times: { type: integer, optional: true, default: 1 }, name: { type: string } } - models: { writer: { keywords: [fast] } } + models: { fast: {} } "; #[test] @@ -46,12 +46,15 @@ fn the_frontmatter_lists_every_declaration_with_arguments_sorted_by_name() -> Te .map(|entry| (entry.vendor(), entry.id().to_string())) .collect(); assert_eq!(packs, [("runvil", "text".to_owned())]); - let roles: Vec<_> = frontmatter - .models() - .iter() - .map(|(label, role)| (label.to_string(), role.min_context())) + let roles: Vec<_> = ModelRole::ALL + .into_iter() + .filter_map(|role| { + frontmatter + .role_requirements(role) + .map(|declared| (role, declared.min_context())) + }) .collect(); - assert_eq!(roles, [("writer".to_owned(), None)]); + assert_eq!(roles, [(ModelRole::Fast, None)]); assert!( frontmatter @@ -71,37 +74,40 @@ fn the_frontmatter_lists_every_declaration_with_arguments_sorted_by_name() -> Te } #[test] -fn a_role_check_by_hand_finds_the_same_shortfall_prepare_reports() -> TestResult { +fn a_role_check_by_hand_finds_the_same_shortfall_the_run_reports() -> TestResult { let source = prompt( - "models: { writer: { keywords: [fast] }, checker: { keywords: [no-thinking], min_context: 100000 } }", + "models: { fast: {}, small: { thinking: false, min_context: 100000 } }", &[("Greet", "return 'done'")], ); let model = model("canned", 200_000, ThinkingMode::Always)?; - let Prepared { - prompt, - requirements, - .. - } = Scenario::default().model(model.clone()).prepare(&source)?; + let scenario = Scenario::default().model(model.clone()); + let Built { prompt, .. } = scenario.build(&source)?; let mut misses = Vec::new(); - for (label, role) in prompt.frontmatter().models() { - let keyword_miss = role.keywords().iter().any(|keyword| match keyword { - ModelKeyword::Thinking => model.thinking() == ThinkingMode::Never, - ModelKeyword::NoThinking => model.thinking() == ThinkingMode::Always, - _ => false, - }); - let context_miss = role.min_context().is_some_and(|min| min > model.context()); - if keyword_miss || context_miss { - misses.push((label, keyword_miss, context_miss)); + for role in ModelRole::ALL { + let Some(declared) = prompt.frontmatter().role_requirements(role) else { + continue; + }; + let thinking_miss = match declared.thinking() { + Some(true) => model.thinking() == ThinkingMode::Never, + Some(false) => model.thinking() == ThinkingMode::Always, + None => false, + }; + let context_miss = declared + .min_context() + .is_some_and(|min| min > model.context()); + if thinking_miss || context_miss { + misses.push((role, thinking_miss, context_miss)); } } - assert_eq!(misses, [("checker", true, false)]); + assert_eq!(misses, [(ModelRole::Small, true, false)]); - let [unmet] = requirements.unmet_models.as_slice() else { - return Err("only the checker falls short".into()); - }; - assert_eq!(unmet.role, "checker"); - assert_eq!(unmet.check, RequirementCheck::HardKeyword); - assert_eq!(unmet.required, "no-thinking"); + assert_eq!( + refusal(scenario, &source)?, + "the environment cannot satisfy this prompt:\n\ + - role 'small': requires 'thinking: false'; \ + the thinking capability of the model filling it is Always", + "only the small role falls short, on its thinking requirement" + ); Ok(()) } diff --git a/crates/runvil/tests/suite/replay.rs b/crates/runvil/tests/suite/replay.rs deleted file mode 100644 index b573d08df..000000000 --- a/crates/runvil/tests/suite/replay.rs +++ /dev/null @@ -1,23 +0,0 @@ -//! Keeping a run's behavior flags for the next run. - -use runvil::replay::Flags; - -use super::support::{Scenario, TestResult, note}; - -#[test] -fn stored_flags_rebuild_into_equal_flags_for_the_next_run() -> TestResult { - let source = note("'hello'"); - let original = Scenario::default().prepare(&source)?.ctx.flags(); - let stored: u32 = original.bits(); - let first = Scenario::default().run(&source)?; - assert_eq!(first.text()?, "hello"); - assert_eq!(stored, 0); - - let restored = Flags::from_bits(stored); - assert_eq!(restored, original); - let next = Scenario::default().context(move |ctx| ctx.with_flags(restored)); - assert_eq!(next.prepare(&source)?.ctx.flags(), original); - let second = next.run(&source)?; - assert_eq!(second.text()?, "hello"); - Ok(()) -} diff --git a/crates/runvil/tests/suite/shipped.rs b/crates/runvil/tests/suite/shipped.rs index 4392ec648..285b9de56 100644 --- a/crates/runvil/tests/suite/shipped.rs +++ b/crates/runvil/tests/suite/shipped.rs @@ -4,7 +4,7 @@ use std::fs; use std::path::Path; -use runvil::effect::{EffectRecord, ToolCaller}; +use runvil::effect::{ToolCallOrigin, ToolCaller}; use runvil::tools::ToolCatalog; use serde_json::{Value, json}; @@ -26,7 +26,7 @@ fn research_ada_lovelace() -> TestResult { [search, text("a factual summary")], ) .args("Ada Lovelace") - .context(move |ctx| ctx.with_tools(catalog.clone())) + .config(move |ctx| ctx.with_tools(catalog.clone())) .run(&fs::read_to_string(path)?) } @@ -37,15 +37,14 @@ fn the_shipped_research_prompt_offers_its_web_tools_by_id_and_runs_end_to_end() let (mut advertised, mut tool_calls) = (Vec::new(), Vec::new()); for logged in &trace.effects { - match &logged.effect { - EffectRecord::Chat { tools, .. } => advertised.push(tools.clone()), - EffectRecord::ToolCall { - tool, - alias, - origin, - .. - } => tool_calls.push((tool.to_string(), alias.clone(), origin.caller)), - _ => {} + if let Some(chat) = logged.effect_of("Chat") { + advertised.push(serde_json::from_value::>( + chat["tools"].clone(), + )?); + } + if let Some(call) = logged.effect_of("ToolCall") { + let origin: ToolCallOrigin = serde_json::from_value(call["origin"].clone())?; + tool_calls.push((call["tool"].clone(), call["alias"].clone(), origin.caller)); } } let wire_names = vec!["web_search".to_owned(), "web_fetch".to_owned()]; @@ -56,11 +55,7 @@ fn the_shipped_research_prompt_offers_its_web_tools_by_id_and_runs_end_to_end() ); assert_eq!( tool_calls, - [( - "web/search".to_owned(), - "web_search".to_owned(), - ToolCaller::Model - )], + [(json!("web/search"), json!("web_search"), ToolCaller::Model)], "the model's call by wire name reaches the tool its id names" ); Ok(()) @@ -71,9 +66,12 @@ fn the_shipped_research_prompt_names_no_tool_by_its_id_in_what_the_model_reads() let trace = research_ada_lovelace()?; let mut checked = 0; for logged in &trace.effects { - let EffectRecord::Chat { messages, .. } = &logged.effect else { + let Some(chat) = logged.effect_of("Chat") else { continue; }; + let messages = chat["messages"] + .as_array() + .ok_or("a chat record holds its messages as a list")?; let user = messages .iter() .filter(|message| message.get("role").and_then(Value::as_str) == Some("user")); diff --git a/crates/runvil/tests/suite/support.rs b/crates/runvil/tests/suite/support.rs index 17e3f9d16..6cdb21dcf 100644 --- a/crates/runvil/tests/suite/support.rs +++ b/crates/runvil/tests/suite/support.rs @@ -16,7 +16,7 @@ use serde_json::Value; mod scenario; -pub(super) use scenario::{Prepared, Scenario, Trace}; +pub(super) use scenario::{Built, Scenario, Trace}; pub(super) type TestResult = Result>; @@ -107,9 +107,8 @@ pub(super) fn shout() -> Tool { } } -/// The model every scenario sets before prepare unless it names its own. -/// It fills only roles a prompt declares, so a prompt without roles -/// ignores it. +/// The model every scenario gives the `default` role unless it names its +/// own, and so the model every other role falls back to. pub(super) fn canned() -> TestResult { model("canned", 8_192, ThinkingMode::Never) } diff --git a/crates/runvil/tests/suite/support/scenario.rs b/crates/runvil/tests/suite/support/scenario.rs index 9b62c1b6b..3d13ffc6b 100644 --- a/crates/runvil/tests/suite/support/scenario.rs +++ b/crates/runvil/tests/suite/support/scenario.rs @@ -4,47 +4,49 @@ //! The driver answers a store effect with `perform_vfs_op` and a timer as //! fired at once, a chat effect with the next scripted reply served under //! the bound model's name, and a tool call with the registered handler. An -//! effect the scenario holds waits, and gets `Dropped` once the run is -//! decided or withdraws it; no other effect is ever dropped. The driver -//! fails the test instead of hanging when the chat script runs out, when -//! scripted replies are left over at `Done`, or when an undecided run -//! issues nothing new while only held effects remain. It also fails a run -//! that reports `Done` while it still holds an effect, because `Done` -//! waits for every issued effect's answer. - -use std::collections::VecDeque; +//! effect the scenario holds waits. An effect the run withdraws gets +//! `Dropped`, whether the scenario holds it or the withdrawing step issued +//! it; no other effect is ever dropped. The driver fails the test instead +//! of hanging when the chat script runs out, when scripted replies are +//! left over at `Done`, or when a run issues and withdraws nothing while +//! only held effects remain. It also fails a run that reports `Done` while +//! it still holds an effect, because `Done` waits for every issued +//! effect's answer. + +use std::collections::{BTreeMap, VecDeque}; use std::sync::Arc; -use runvil::effect::{AnswerRecord, Effect, EffectAnswer, EffectId, EffectRecord, IssuedEffect}; +use runvil::effect::{Effect, EffectAnswer, EffectId, IssuedEffect}; use runvil::event::Event; use runvil::ids::Provenance; -use runvil::model::{Completion, ModelInfo}; +use runvil::model::{Completion, ModelInfo, ModelRole}; use runvil::timestamp::Timestamp; use runvil::tools::ToolCatalog; use runvil::vfs::perform_vfs_op; -use runvil::{Prompt, Requirements, Run, RunContext, RunResult, RunSeed, Step}; -use serde_json::json; +use runvil::{Prompt, Run, RunConfig, RunOutcome, RunSeed, Step}; +use serde_json::{Value, json}; use super::{Reply, TestResult, Tool, canned, tool_info}; -type Edit = Box RunContext>; +type Edit = Box RunConfig>; /// The seed every scenario's run takes. const TEST_SEED: RunSeed = RunSeed::from_bytes([7; 32]); -/// One test's setup. Every scenario sets [`canned`] unless it names a -/// model, numbers run events on from the parse events, and fails to start -/// when prepare reports a gap. +/// One test's setup. Every scenario gives the `default` role [`canned`] +/// unless it names a model or a role map, and numbers run events on from +/// the parse events. #[derive(Default)] pub(crate) struct Scenario { args: String, started_at: Timestamp, model: Option, + roles: Option>, tool: Option, replies: Vec, hold: Option bool>, reverse: bool, - context: Option, + config: Option, } impl Scenario { @@ -66,8 +68,8 @@ impl Scenario { } } - /// Holds each effect `hold` picks, unanswered, until the run is decided - /// or withdraws it. + /// Holds each effect `hold` picks, unanswered, until the run withdraws + /// it. pub(crate) fn hold(mut self, hold: fn(&Effect) -> bool) -> Self { self.hold = Some(hold); self @@ -85,65 +87,66 @@ impl Scenario { self } - /// Sets `model` as the run's current model, in place of [`canned`]. + /// Sets `model` as the `default` role's model, in place of [`canned`]. pub(crate) fn model(mut self, model: ModelInfo) -> Self { self.model = Some(model); self } + /// Sets the whole role map the run fills the roles from, in place of the + /// one-model map [`model`](Self::model) and [`canned`] make. The map may + /// leave `default` out. + pub(crate) fn roles(mut self, roles: impl IntoIterator) -> Self { + self.roles = Some(roles.into_iter().collect()); + self + } + /// Answers each batch of effects in reverse issue order. pub(crate) fn reverse(mut self) -> Self { self.reverse = true; self } - /// Edits the run's context after the scenario sets its own, so an edit + /// Edits the run's config after the scenario sets its own, so an edit /// can also undo one of them. - pub(crate) fn context(mut self, edit: impl Fn(RunContext) -> RunContext + 'static) -> Self { - self.context = Some(Box::new(edit)); + pub(crate) fn config(mut self, edit: impl Fn(RunConfig) -> RunConfig + 'static) -> Self { + self.config = Some(Box::new(edit)); self } - /// Parses `source` as the run `greeter` and prepares its context, - /// whatever prepare reports. [`start`](Self::start) is what fails on - /// a gap. - pub(crate) fn prepare(&self, source: &str) -> TestResult { + /// Parses `source` as the run `greeter` and builds its config. + pub(crate) fn build(&self, source: &str) -> TestResult { let (parsed, events) = Prompt::parse(source, "greeter"); let prompt = parsed?; - let model = match &self.model { - Some(model) => model.clone(), - None => canned()?, + let roles = match (&self.roles, &self.model) { + (Some(roles), _) => roles.clone(), + (None, Some(model)) => BTreeMap::from([(ModelRole::Default, model.clone())]), + (None, None) => BTreeMap::from([(ModelRole::Default, canned()?)]), }; - let mut ctx = RunContext::new("greeter", TEST_SEED, self.started_at) - .with_provenance_start(u32::try_from(events.len())?); + let mut ctx = RunConfig::new("greeter", TEST_SEED, self.started_at) + .with_provenance_start(u32::try_from(events.len())?) + .with_models(roles); if let Some(tool) = &self.tool { let info = tool_info(tool.id, json!({ "type": "object" }))?; ctx = ctx.with_tools(ToolCatalog::new([info])?); } - if let Some(edit) = &self.context { + if let Some(edit) = &self.config { ctx = edit(ctx); } - let (ctx, requirements) = ctx.prepare(&prompt, Some(model)); - Ok(Prepared { + Ok(Built { prompt, ctx, - requirements, events, }) } - /// Prepares `source` and creates its run, not yet stepped. Fails when - /// prepare reports a gap. + /// Builds `source`'s config and creates its run, not yet stepped. pub(crate) fn start(self, source: &str) -> TestResult { - let Prepared { + let Built { prompt, ctx, - requirements, events, - } = self.prepare(source)?; - if let Some(refusal) = requirements.refusal() { - return Err(refusal.into()); - } + } = self.build(source)?; Ok(Driver { run: Run::new(Arc::new(prompt), &self.args, ctx), replies: self.replies.into(), @@ -153,7 +156,7 @@ impl Scenario { held: Vec::new(), events, effects: Vec::new(), - result: None, + outcome: None, }) } @@ -162,10 +165,9 @@ impl Scenario { } } -pub(crate) struct Prepared { +pub(crate) struct Built { pub(crate) prompt: Prompt, - pub(crate) ctx: RunContext, - pub(crate) requirements: Requirements, + pub(crate) ctx: RunConfig, /// The parse events. pub(crate) events: Vec, } @@ -180,20 +182,20 @@ pub(crate) struct Driver { held: Vec, events: Vec, effects: Vec, - result: Option, + outcome: Option, } struct Issued { id: EffectId, provenance: Provenance, - effect: EffectRecord, + effect: Value, } impl Driver { /// Steps the run once, answers or holds each effect the step issued, - /// and drops the held effects it may. Returns how many it issued. + /// and drops each effect the step withdrew. Returns how many it issued. pub(crate) fn pump(&mut self) -> TestResult { - if self.result.is_some() { + if self.outcome.is_some() { return Err("the run is done, so there is nothing left to step".into()); } let (mut effects, withdrawn) = match self.run.step() { @@ -205,7 +207,7 @@ impl Driver { self.events.extend(events); (effects, withdrawn) } - Step::Done { result, events } => { + Step::Done { outcome, events } => { self.events.extend(events); let held = self.held.len(); if held > 0 { @@ -215,7 +217,7 @@ impl Driver { if left > 0 { return Err(format!("the run ended with {left} scripted replies left").into()); } - self.result = Some(result); + self.outcome = Some(outcome); return Ok(0); } }; @@ -236,17 +238,18 @@ impl Driver { provenance, effect: effect_record, }; - if self.hold.is_some_and(|hold| hold(&effect)) { + if withdrawn.contains(&id) { + self.respond(held, EffectAnswer::Dropped); + } else if self.hold.is_some_and(|hold| hold(&effect)) { self.held.push(held); } else { let answer = self.answer(effect)?; self.respond(held, answer); } } - let decided = self.run.decided(); let (dropped, held): (Vec<_>, Vec<_>) = std::mem::take(&mut self.held) .into_iter() - .partition(|issued| decided || withdrawn.contains(&issued.id)); + .partition(|issued| withdrawn.contains(&issued.id)); self.held = held; for issued in dropped { self.respond(issued, EffectAnswer::Dropped); @@ -254,6 +257,11 @@ impl Driver { Ok(issued) } + /// How many effects the driver holds unanswered. + pub(crate) fn held(&self) -> usize { + self.held.len() + } + fn respond(&mut self, issued: Issued, answer: EffectAnswer) { self.effects.push(Logged { provenance: issued.provenance, @@ -266,7 +274,7 @@ impl Driver { pub(crate) fn pump_until_held(&mut self) -> TestResult { while self.held.is_empty() { self.advance()?; - if self.result.is_some() { + if self.outcome.is_some() { return Err("the run finished without issuing an effect to hold".into()); } } @@ -275,9 +283,9 @@ impl Driver { pub(crate) fn finish(mut self) -> TestResult { loop { - if let Some(result) = self.result.take() { + if let Some(outcome) = self.outcome.take() { return Ok(Trace { - result, + outcome, events: self.events, effects: self.effects, }); @@ -286,14 +294,15 @@ impl Driver { } } - /// Pumps once, and fails when another step could only wait forever. + /// Pumps once, and fails when another step could only wait forever. A + /// step that issues nothing can only withdraw held effects, so an + /// unchanged count means it withdrew nothing. fn advance(&mut self) -> TestResult { let held = self.held.len(); let issued = self.pump()?; - let waiting = self.result.is_none() && !self.run.decided(); - if waiting && issued == 0 && self.held.len() == held { + if self.outcome.is_none() && issued == 0 && self.held.len() == held { return Err(format!( - "the undecided run issued nothing new while only {held} held effects remain" + "the run issued and withdrew nothing while only {held} held effects remain" ) .into()); } @@ -333,7 +342,7 @@ impl Driver { } pub(crate) struct Trace { - pub(crate) result: RunResult, + pub(crate) outcome: RunOutcome, /// The parse events, then the run's events, in report order. pub(crate) events: Vec, /// Every effect the run issued, in the order the driver answered them. @@ -341,17 +350,25 @@ pub(crate) struct Trace { } impl Trace { - /// The text of a `RunResult::Completed`. Fails on any other result. + /// The text of a `RunOutcome::Completed`. Fails on any other outcome. pub(crate) fn text(&self) -> TestResult<&str> { - match &self.result { - RunResult::Completed(text) => Ok(text), - result => Err(format!("expected RunResult::Completed, got {result:?}").into()), + match &self.outcome { + RunOutcome::Completed { final_text } => Ok(final_text), + outcome => Err(format!("expected RunOutcome::Completed, got {outcome:?}").into()), } } } pub(crate) struct Logged { pub(crate) provenance: Provenance, - pub(crate) effect: EffectRecord, - pub(crate) answer: AnswerRecord, + pub(crate) effect: Value, + pub(crate) answer: Value, +} + +impl Logged { + /// The effect record's body when the effect is of `kind`, such as + /// `Chat` or `ToolCall`. + pub(crate) fn effect_of(&self, kind: &str) -> Option<&Value> { + self.effect.get(kind) + } } diff --git a/crates/runvil/tests/suite/timestamp.rs b/crates/runvil/tests/suite/timestamp.rs index 643e77146..ba0031014 100644 --- a/crates/runvil/tests/suite/timestamp.rs +++ b/crates/runvil/tests/suite/timestamp.rs @@ -5,7 +5,7 @@ use runvil::timestamp::Timestamp; use super::support::{Scenario, TestResult, note}; #[test] -fn the_prompt_reads_the_contexts_start_time_as_sys_when() -> TestResult { +fn the_prompt_reads_the_configs_start_time_as_sys_when() -> TestResult { let source = note("sys.when"); let stamp = Timestamp::from_unix_millis(1_700_000_000_000); diff --git a/crates/runvil/tests/suite/vfs.rs b/crates/runvil/tests/suite/vfs.rs index 42fc148d6..143836fbc 100644 --- a/crates/runvil/tests/suite/vfs.rs +++ b/crates/runvil/tests/suite/vfs.rs @@ -10,7 +10,7 @@ use runvil::vfs::{ use super::support::{Scenario, TestResult, note}; fn run_note(vfs: VfsRef) -> TestResult { - let scenario = Scenario::default().context(move |ctx| ctx.with_vfs(vfs.clone())); + let scenario = Scenario::default().config(move |ctx| ctx.with_vfs(vfs.clone())); Ok(scenario.run(¬e("'hello'"))?.text()?.to_owned()) } diff --git a/crates/workshop/agents/agents/chat.md b/crates/workshop/agents/agents/chat.md index 6b6f8a427..f2c7566bd 100644 --- a/crates/workshop/agents/agents/chat.md +++ b/crates/workshop/agents/agents/chat.md @@ -3,30 +3,23 @@ name: chat description: The built-in Workshop chat agent on the unified runtime. runvil: 0 uses: - - web - user-input + - std + - web models: - chat: + default: min_context: 32768 - description: "A conversational frontier model suitable for long-lived operator sessions" --- # Chat -The built-in chat agent: a transparent pass-through between the operator -and the selected model. The conversation is the section's `messages` list, -retained across turns; the model is the dropdown's selection bound at run -launch, so a selection change takes effect on the next run. When the -conversation outgrows the model's context window, older turns are -summarized so the conversation continues. - -```lua -models.default("chat") -``` +The built-in chat agent: a pass-through between the operator and the selected model, whose one system message says where the Runvil prompting language documentation is mounted. The conversation is the section's `messages` list, retained across turns; the model is the dropdown's selection bound at run launch, so a selection change takes effect on the next run. When the conversation outgrows the model's context window, older turns are summarized so the conversation continues. ## Conversation ```lua +tools.offer("user-input") +tools.offer("std") tools.offer("web") tools.offer(tools.packs.extras()) local function compact(reason) @@ -58,6 +51,7 @@ local function compact(reason) meta = { summary = true }, }) end +--messages:system("Runvil prompting language documentation can be found in /_runvil/language-guide") while true do local text = input.ask() messages:user(text) diff --git a/crates/workshop/agents/src/conversation-run.rs b/crates/workshop/agents/src/conversation-run.rs index edb714bda..6505367e9 100644 --- a/crates/workshop/agents/src/conversation-run.rs +++ b/crates/workshop/agents/src/conversation-run.rs @@ -75,9 +75,7 @@ impl Conversation { fn take_report(&self, result: Result) { let message = match result { Ok(report) => { - if let Some(run) = report.run_id { - self.note_run(run); - } + self.note_run(report.run_id); match report.outcome { RunOutcome::Completed { .. } => return, RunOutcome::Cancelled => { diff --git a/crates/workshop/agents/src/input-tests.rs b/crates/workshop/agents/src/input-tests.rs index 7a9a1f788..a09ec692b 100644 --- a/crates/workshop/agents/src/input-tests.rs +++ b/crates/workshop/agents/src/input-tests.rs @@ -309,9 +309,8 @@ async fn an_ask_is_answered_when_the_registry_receives_the_text() { }, "the operator's text is the section's return value, byte-exact" ); - let run = report.run_id.expect("the recorder began the run"); assert_eq!( - recorder.outcome(run), + recorder.outcome(report.run_id), Some(report.outcome), "the recorder holds the operator's text in the run's outcome" ); diff --git a/crates/workshop/agents/tests/it/conversation.rs b/crates/workshop/agents/tests/it/conversation.rs index a1f647129..1a19cba6a 100644 --- a/crates/workshop/agents/tests/it/conversation.rs +++ b/crates/workshop/agents/tests/it/conversation.rs @@ -161,7 +161,7 @@ async fn a_close_with_a_question_open_cancels_it_and_reports_the_interrupt() { } #[tokio::test] -async fn a_close_before_the_run_starts_ends_it_with_no_run() { +async fn a_close_before_the_run_starts_ends_its_recorded_run_cancelled() { let table = Conversations::new(); let conversation = table.open("early"); let recorder = Arc::new(MemoryRecorder::new()); @@ -170,10 +170,11 @@ async fn a_close_before_the_run_starts_ends_it_with_no_run() { conversation.close(); ended(start(&conversation, &recorder, "return 'never'")).await; - assert_eq!( - conversation.run_id(), - None, - "the recorder never began a run" + let run = conversation.run_id().expect("the recorder began the run"); + assert_eq!(recorder.outcome(run), Some(RunOutcome::Cancelled)); + assert!( + recorder.records(run).is_empty(), + "the run ended before its prompt was parsed" ); assert_eq!( reports(&mut errors) diff --git a/crates/workshop/run-log/src/append.rs b/crates/workshop/run-log/src/append.rs index bdd188d9a..07fda9335 100644 --- a/crates/workshop/run-log/src/append.rs +++ b/crates/workshop/run-log/src/append.rs @@ -139,7 +139,9 @@ impl RunLog { let kind = outcome.as_str(); let (final_text, error_kind, error_message) = match outcome { RunOutcome::Completed { final_text } => (Some(final_text), None, None), - RunOutcome::Failed { kind, message } => (None, Some(kind), Some(message)), + RunOutcome::Failed { kind, message } => { + (None, Some(kind.as_str().to_owned()), Some(message)) + } RunOutcome::Cancelled => (None, None, None), }; let changed = self diff --git a/crates/workshop/run-log/src/lib.rs b/crates/workshop/run-log/src/lib.rs index fb258607d..47d849e2a 100644 --- a/crates/workshop/run-log/src/lib.rs +++ b/crates/workshop/run-log/src/lib.rs @@ -55,6 +55,7 @@ mod schema; pub use append::RunLog; pub use error::LogError; pub use record::{ - Record, RecordFilter, RecordKind, RunId, RunMeta, RunOutcome, RunRow, Seq, StoredRecord, + Record, RecordFilter, RecordKind, RunErrorKind, RunId, RunMeta, RunOutcome, RunRow, Seq, + StoredRecord, }; pub use recorder::{AgentRecorder, TursoRecorder}; diff --git a/crates/workshop/run-log/src/read.rs b/crates/workshop/run-log/src/read.rs index a976e2d2c..ee5e986c5 100644 --- a/crates/workshop/run-log/src/read.rs +++ b/crates/workshop/run-log/src/read.rs @@ -5,7 +5,8 @@ use harness::record::RunSeed; use crate::append::{RunLog, unsigned}; use crate::error::LogError; use crate::record::{ - Record, RecordFilter, RecordKind, RunId, RunMeta, RunOutcome, RunRow, Seq, StoredRecord, + Record, RecordFilter, RecordKind, RunErrorKind, RunId, RunMeta, RunOutcome, RunRow, Seq, + StoredRecord, }; use crate::schema; @@ -126,6 +127,10 @@ fn to_seed(text: String) -> Result { } /// Rebuilds a [`RunOutcome`] from its four columns. +/// +/// A failed run's kind reads back through [`RunErrorKind::from_tag`]. A +/// stored tag the Engine does not name, such as the `Input` that older +/// rows hold, reads as [`RunErrorKind::Internal`] with its stored message. fn parse_outcome( kind: &str, final_text: Option, @@ -134,7 +139,10 @@ fn parse_outcome( ) -> Result { match (kind, final_text, error_kind, error_message) { ("completed", Some(final_text), None, None) => Ok(RunOutcome::Completed { final_text }), - ("failed", None, Some(kind), Some(message)) => Ok(RunOutcome::Failed { kind, message }), + ("failed", None, Some(kind), Some(message)) => Ok(RunOutcome::Failed { + kind: RunErrorKind::from_tag(&kind).unwrap_or(RunErrorKind::Internal), + message, + }), ("cancelled", None, None, None) => Ok(RunOutcome::Cancelled), (kind, final_text, error_kind, error_message) => Err(LogError::Corrupt(format!( "expected outcome columns to match {kind:?}, found final_text={final_text:?} \ diff --git a/crates/workshop/run-log/src/record.rs b/crates/workshop/run-log/src/record.rs index f0a2ce2d4..10c67c1c1 100644 --- a/crates/workshop/run-log/src/record.rs +++ b/crates/workshop/run-log/src/record.rs @@ -1,13 +1,12 @@ //! The values the run log stores and returns. //! //! The values the Harness hands a recorder (`Record`, `RunMeta`, -//! `RunOutcome`, and their parts) are the Harness's own, re-exported from -//! `harness::record`, so a log and the effect loop share one definition -//! of each. +//! `RunOutcome`, and their parts) are re-exported from `harness::record`, +//! so a log and the effect loop share one definition of each. use std::fmt; -pub use harness::record::{Record, RecordKind, RunId, RunMeta, RunOutcome}; +pub use harness::record::{Record, RecordKind, RunErrorKind, RunId, RunMeta, RunOutcome}; /// A record's position in its run: the effect loop's order, assigned by /// the log in call order, starting at `0` and strictly increasing. diff --git a/crates/workshop/run-log/tests/it/append.rs b/crates/workshop/run-log/tests/it/append.rs index 58485c978..48454668a 100644 --- a/crates/workshop/run-log/tests/it/append.rs +++ b/crates/workshop/run-log/tests/it/append.rs @@ -3,7 +3,7 @@ use harness::record::RunSeed; use serde_json::json; use workshop_run_log::{ - LogError, Record, RecordFilter, RecordKind, RunId, RunLog, RunMeta, RunOutcome, + LogError, Record, RecordFilter, RecordKind, RunErrorKind, RunId, RunLog, RunMeta, RunOutcome, }; /// Every record of a run, in loop order. @@ -219,7 +219,7 @@ async fn a_failed_outcome_keeps_its_kind_and_message() { let mut log = RunLog::in_memory().await.unwrap(); let run = log.begin_run(meta(), AGENT).await.unwrap(); let outcome = RunOutcome::Failed { - kind: "tool".to_owned(), + kind: RunErrorKind::Tool, message: "expected a reply, got nothing".to_owned(), }; log.end_run(run, outcome.clone()).await.unwrap(); @@ -233,6 +233,66 @@ async fn a_failed_outcome_keeps_its_kind_and_message() { ); } +#[tokio::test] +async fn a_stored_kind_the_engine_does_not_name_reads_as_internal_with_its_message() { + // Each run ends as a `RequirementsUnmet` failure, and its stored kind + // is then set as an older build may have written it. + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("runs.db"); + let message = "the input file could not be put in place"; + for (stored, read) in [ + ("Vfs", RunErrorKind::Vfs), + ("Input", RunErrorKind::Internal), + ] { + let run = { + let mut log = RunLog::open(&path).await.unwrap(); + let run = log.begin_run(meta(), AGENT).await.unwrap(); + let failed = RunOutcome::Failed { + kind: RunErrorKind::RequirementsUnmet, + message: message.to_owned(), + }; + log.end_run(run, failed).await.unwrap(); + run + }; + let raw = turso::Builder::new_local(path.to_str().unwrap()) + .build() + .await + .unwrap() + .connect() + .unwrap(); + let mut rows = raw + .query( + "SELECT error_kind FROM runs WHERE run_id = ?1", + (run.get(),), + ) + .await + .unwrap(); + let written: String = rows.next().await.unwrap().unwrap().get(0).unwrap(); + assert_eq!( + written, "RequirementsUnmet", + "the column holds the kind's tag" + ); + drop(rows); + raw.execute( + "UPDATE runs SET error_kind = ?2 WHERE run_id = ?1", + (run.get(), stored), + ) + .await + .unwrap(); + drop(raw); + + let log = RunLog::open(&path).await.unwrap(); + assert_eq!( + log.run(run).await.unwrap().outcome, + Some(RunOutcome::Failed { + kind: read, + message: message.to_owned(), + }), + "the stored tag {stored:?} reads as {read:?} and keeps its message" + ); + } +} + #[tokio::test] async fn a_run_ends_exactly_once() { let mut log = RunLog::in_memory().await.unwrap(); diff --git a/crates/workshop/run-log/tests/it/fidelity.rs b/crates/workshop/run-log/tests/it/fidelity.rs index 31a985f12..a7abe1222 100644 --- a/crates/workshop/run-log/tests/it/fidelity.rs +++ b/crates/workshop/run-log/tests/it/fidelity.rs @@ -7,7 +7,7 @@ use harness::record::RunSeed; use runvil::event::{Event, ReplyOrigin}; -use runvil::ids::{ParseIdError, Provenance, RoundId, TaskId}; +use runvil::ids::{IdError, Provenance, RoundId, TaskId}; use runvil::metrics::{CallMetrics, ClientTiming, LlamaTimings, Usage, VllmMetrics}; use serde_json::{Map, Value, json}; use workshop_run_log::{Record, RecordFilter, RecordKind, RunLog, RunMeta}; @@ -34,7 +34,7 @@ fn meta() -> RunMeta { } /// The replay key every fidelity event carries. -fn provenance() -> Result { +fn provenance() -> Result { Ok(Provenance::new("0".parse::()?, 0)) } diff --git a/crates/workshop/server/src/agents.rs b/crates/workshop/server/src/agents.rs index c88e66bf5..0f44b709b 100644 --- a/crates/workshop/server/src/agents.rs +++ b/crates/workshop/server/src/agents.rs @@ -15,7 +15,10 @@ //! the built-in subagents, and the conversation's own services, which //! hold its input broker. The Harness reaches the gateway //! only through that broker, which follows the live gateway binding and -//! sends each round to the dropdown's current pick. The +//! sends each round to the dropdown's current pick. Each run's +//! filesystem is an in-memory root, with the language guide demo's +//! read-only mount beside it while that demo's switch is on +//! (`language_guide`). The //! run's request carries the Host snapshot ([`bindings`]): the menu's //! pick and the workspace's granted roots, read once the menu's catalog //! holds a chat-capable model. Status-bar reporting stays in the server @@ -31,6 +34,7 @@ mod bindings; mod broker; mod gateway; +mod language_guide; mod relay; mod search; mod socket; @@ -46,7 +50,6 @@ use std::sync::Arc; use harness::pack::{HostContext, HostServices, PackRegistry}; use harness::record::RunId; -use harness::vfs::VfsRef; use harness::{Harness, RunRequest}; use plugin_web::SEARCH_PROVIDER; use runvil::tools::ToolId; @@ -326,7 +329,7 @@ impl AgentSessions { source, args: String::new(), input_text: None, - vfs: VfsRef::default(), + vfs: language_guide::run_vfs(), host: host_snapshot(&self.inner.registry), }; conversation diff --git a/crates/workshop/server/src/agents/language_guide.rs b/crates/workshop/server/src/agents/language_guide.rs new file mode 100644 index 000000000..8512b0475 --- /dev/null +++ b/crates/workshop/server/src/agents/language_guide.rs @@ -0,0 +1,55 @@ +//! A demo of the virtual filesystem: the Runvil prompting language guide, +//! mounted read-only in every run at [`MOUNT`], so an agent can read the +//! language's own documentation with the file tools. +//! +//! The guide's pages are read from [`SOURCE`], a directory on one +//! machine, so this is a hack, and it is all here. [`ENABLED`] is its one +//! switch: `true` mounts the guide in each run, and `false` leaves each +//! run its plain in-memory root. The chat agent's system prompt, which +//! names the mount, is in `chat.md` and is toggled there. + +use std::path::Path; + +use runvil::vfs::{RealBackend, VfsRef, Writability}; + +/// The switch: `true` mounts the guide in every run. +const ENABLED: bool = false; + +/// The directory on this machine that holds the guide's pages. +const SOURCE: &str = r"C:\Users\Vinnie\cursor\runvil-docs\src\language"; + +/// The virtual path a run sees the guide at. +const MOUNT: &str = "/_runvil/language-guide"; + +/// The filesystem for one run: the in-memory root, with the guide mounted +/// at [`MOUNT`] when [`ENABLED`]. +pub(super) fn run_vfs() -> VfsRef { + let base = VfsRef::default(); + if ENABLED { + mount_at(base, MOUNT, Path::new(SOURCE)) + } else { + base + } +} + +/// `base` with the directory `source` mounted read-only at `mount`, an +/// absolute virtual path. A `source` that cannot be mounted leaves `base` +/// as it is and logs a warning, because a run without the mount still +/// works. +fn mount_at(base: VfsRef, mount: &str, source: &Path) -> VfsRef { + match RealBackend::rooted(source) { + Ok(backend) => base.overlay(mount, backend.with_writability(Writability::ReadOnly)), + Err(error) => { + tracing::warn!( + source = %source.display(), + mount, + %error, + "a directory is not mounted; runs start without it" + ); + base + } + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/workshop/server/src/agents/language_guide/tests.rs b/crates/workshop/server/src/agents/language_guide/tests.rs new file mode 100644 index 000000000..79e6174bb --- /dev/null +++ b/crates/workshop/server/src/agents/language_guide/tests.rs @@ -0,0 +1,171 @@ +//! Mapping a local directory into a run's filesystem: a directory mapped +//! under `/_runvil` reads through the mapping, is reached by listing down +//! from the root, refuses writes, leaves the in-memory root writable, and +//! is visible to a prompt that runs over the filesystem. Every test uses a +//! local directory it makes, so none depends on the language guide's files +//! or on a path on any one machine. + +use std::error::Error; +use std::path::Path; +use std::sync::Arc; + +use harness::record::RunOutcome; +use runvil::vfs::{Access, FileType, Origin, VfsError, VfsRef}; +use workshop_registry::Registry; + +use super::mount_at; +use crate::agents::host_context; +use crate::agents::tests::run_prompt_over; + +/// What a test returns: any error from setup or from the VFS fails it. +type Outcome = Result<(), Box>; + +/// Where the local directory is mapped. +const MOUNT: &str = "/_runvil/test"; + +/// The one page the local directory holds. +const PAGE: &str = "hello.md"; + +/// What that page says. +const TEXT: &str = "# Hello from a local page\n"; + +/// A directory holding the one page. +fn local_directory() -> Result> { + let directory = tempfile::tempdir()?; + std::fs::write(directory.path().join(PAGE), TEXT)?; + Ok(directory) +} + +/// The filesystem with `source` mapped to [`MOUNT`]. +fn filesystem_with(source: &Path) -> VfsRef { + mount_at(VfsRef::default(), MOUNT, source) +} + +/// A run's access over the filesystem with `source` mapped to [`MOUNT`]. +fn access_with(source: &Path) -> Result { + filesystem_with(source).acquire(Origin::new("local mapping test")) +} + +#[test] +fn a_page_of_the_directory_reads_through_the_mapping() -> Outcome { + let directory = local_directory()?; + let access = access_with(directory.path())?; + + assert_eq!(access.read_string(&format!("{MOUNT}/{PAGE}"))?, TEXT); + let names: Vec = access + .list(MOUNT)? + .into_iter() + .map(|entry| entry.name) + .collect(); + assert_eq!(names, [PAGE]); + Ok(()) +} + +#[test] +fn listing_down_from_the_root_reaches_the_directory() -> Outcome { + let directory = local_directory()?; + let access = access_with(directory.path())?; + let names = |path: &str| -> Result, VfsError> { + Ok(access + .list(path)? + .into_iter() + .map(|entry| entry.name) + .collect()) + }; + + // The in-memory root holds nothing, so the mapping is the only way down. + assert_eq!(names("/")?, ["_runvil"]); + assert_eq!(names("/_runvil")?, ["test"]); + assert_eq!(names(MOUNT)?, [PAGE]); + assert_eq!(access.stat("/_runvil")?.file_type, FileType::Directory); + Ok(()) +} + +#[test] +fn the_mapping_refuses_a_write_and_leaves_the_page_alone() -> Outcome { + let directory = local_directory()?; + let access = access_with(directory.path())?; + + let created = access.write(&format!("{MOUNT}/new.md"), b"new"); + assert!( + matches!(created, Err(VfsError::ReadOnly { .. })), + "{created:?}" + ); + let replaced = access.write(&format!("{MOUNT}/{PAGE}"), b"changed"); + assert!( + matches!(replaced, Err(VfsError::ReadOnly { .. })), + "{replaced:?}" + ); + assert_eq!(std::fs::read_to_string(directory.path().join(PAGE))?, TEXT); + assert!(!directory.path().join("new.md").exists()); + Ok(()) +} + +#[test] +fn the_in_memory_root_stays_writable_beside_the_mapping() -> Outcome { + let directory = local_directory()?; + let access = access_with(directory.path())?; + + access.write("/scratch.txt", b"kept")?; + assert_eq!(access.read_string("/scratch.txt")?, "kept"); + Ok(()) +} + +#[test] +fn a_directory_that_cannot_be_mapped_leaves_the_run_its_root() -> Outcome { + let directory = local_directory()?; + let missing = directory.path().join("no-such-directory"); + let access = access_with(&missing)?; + + assert!(!access.exists(MOUNT)?, "nothing is mapped"); + access.write("/scratch.txt", b"kept")?; + assert_eq!(access.read_string("/scratch.txt")?, "kept"); + Ok(()) +} + +/// A prompt that lists `/_runvil` and reads the mapped page with the `std` +/// file tools, and returns both answers. +const PROMPT: &str = r"--- +name: sees-the-mapping +description: lists and reads a mapped directory +runvil: 0 +uses: + - std +--- + +# Sees the mapping + +## Only + +```lua +local listing = tools.call('std/dir', { target_directory = '/_runvil' }) +local page = tools.call('std/read', { path = '/_runvil/test/hello.md' }) +return listing .. '\n' .. page +``` +"; + +#[tokio::test] +async fn a_prompt_sees_the_directory_mapped_under_the_runvil_mount() -> Outcome { + let directory = local_directory()?; + let (host, _ask) = host_context(&Registry::new(), None); + + let outcome = run_prompt_over( + &Arc::new(host), + PROMPT.to_owned(), + filesystem_with(directory.path()), + ) + .await; + + let Some(RunOutcome::Completed { final_text }) = outcome else { + return Err(format!("the prompt did not complete: {outcome:?}").into()); + }; + assert!( + final_text.contains(" - test/"), + "the prompt lists the mapping under /_runvil: {final_text}" + ); + assert!( + final_text.contains(" 1|# Hello from a local page"), + "the prompt reads the mapped page: {final_text}" + ); + Ok(()) +} diff --git a/crates/workshop/server/src/agents/tests.rs b/crates/workshop/server/src/agents/tests.rs index ecf67f776..0373bce34 100644 --- a/crates/workshop/server/src/agents/tests.rs +++ b/crates/workshop/server/src/agents/tests.rs @@ -15,7 +15,7 @@ use std::sync::Arc; use std::time::Duration; use harness::pack::{HostContext, HostServices}; -use harness::record::{MemoryRecorder, RunOutcome}; +use harness::record::{MemoryRecorder, RunErrorKind, RunOutcome}; use harness::vfs::VfsRef; use harness::{BoxFuture, Harness, HostSnapshot, InferenceBroker, RunRequest}; use harness_gateway_client::{CompletionError, CompletionErrorKind}; @@ -68,9 +68,20 @@ impl InferenceBroker for OfflineBroker { } } -/// Runs the prompt `source` as one conversation over `host`, then reads -/// the conversation's state and the outcome of its one run. +/// Runs the prompt `source` as one conversation over `host` and the +/// default in-memory filesystem, then reads the conversation's state and +/// the outcome of its one run. async fn run_prompt(host: &Arc, source: String) -> Option { + run_prompt_over(host, source, VfsRef::default()).await +} + +/// Runs the prompt `source` as [`run_prompt`] does, over the filesystem +/// `vfs`. +pub(super) async fn run_prompt_over( + host: &Arc, + source: String, + vfs: VfsRef, +) -> Option { let recorder = Arc::new(MemoryRecorder::new()); let conversation = Conversations::new().open("browses"); let harness = Harness::new( @@ -85,7 +96,7 @@ async fn run_prompt(host: &Arc, source: String) -> Option, /// The minimum context window in tokens; `null` when absent. min_context: Option, - /// The role's prose description; `null` when absent. - description: Option, + /// The thinking requirement; `null` when absent. + thinking: Option, } impl ModelDto { - /// Builds the DTO for the role declared under `label`. - fn new(label: &str, role: &ModelRole) -> Self { + /// Builds the DTO for the requirements declared for `role`. + fn new(role: ModelRole, declared: RoleRequirements) -> Self { Self { - label: label.to_owned(), - keywords: role.keywords().iter().copied().map(keyword_wire).collect(), - min_context: role.min_context().map(std::num::NonZeroU32::get), - description: role.description().map(str::to_owned), + label: role.as_str().to_owned(), + min_context: declared.min_context().map(std::num::NonZeroU32::get), + thinking: declared.thinking(), } } } -/// The kebab-case wire form of a model keyword. -fn keyword_wire(keyword: ModelKeyword) -> &'static str { - match keyword { - ModelKeyword::Thinking => "thinking", - ModelKeyword::NoThinking => "no-thinking", - ModelKeyword::Frontier => "frontier", - ModelKeyword::Fast => "fast", - ModelKeyword::Small => "small", - ModelKeyword::Creative => "creative", - ModelKeyword::Chat => "chat", - // Any other keyword has no wire form of its own. - _ => "unknown", - } -} - impl From<&Frontmatter> for ContractResponse { fn from(frontmatter: &Frontmatter) -> Self { Self { @@ -199,10 +182,13 @@ impl From<&Frontmatter> for ContractResponse { output: frontmatter.output().map(FileDto::from), uses: frontmatter.uses().iter().map(PackDto::from).collect(), args: ArgsDto::from(frontmatter.args()), - models: frontmatter - .models() - .iter() - .map(|(label, role)| ModelDto::new(label, role)) + models: ModelRole::ALL + .into_iter() + .filter_map(|role| { + frontmatter + .role_requirements(role) + .map(|declared| ModelDto::new(role, *declared)) + }) .collect(), } } diff --git a/crates/workshop/server/tests/it/agents.rs b/crates/workshop/server/tests/it/agents.rs index 52b93ebb0..a01aa6486 100644 --- a/crates/workshop/server/tests/it/agents.rs +++ b/crates/workshop/server/tests/it/agents.rs @@ -50,7 +50,7 @@ while true do return end messages:user(text) - models.get('test-model'):loop() + models.loop(models.get('test-model')) end ``` "; diff --git a/crates/workshop/server/tests/it/agents/revoke.rs b/crates/workshop/server/tests/it/agents/revoke.rs index 3e7bc4edf..57f63575a 100644 --- a/crates/workshop/server/tests/it/agents/revoke.rs +++ b/crates/workshop/server/tests/it/agents/revoke.rs @@ -25,11 +25,11 @@ uses: ```lua messages:user('launch@' .. tostring(ui().workspace_root)) -models.get('test-model'):loop() +models.loop(models.get('test-model')) while true do local text = input.ask() messages:user(text .. '@' .. tostring(ui().workspace_root)) - models.get('test-model'):loop() + models.loop(models.get('test-model')) end ``` "; diff --git a/crates/workshop/server/tests/it/chat_gate.rs b/crates/workshop/server/tests/it/chat_gate.rs index 63bad00db..f4684d55c 100644 --- a/crates/workshop/server/tests/it/chat_gate.rs +++ b/crates/workshop/server/tests/it/chat_gate.rs @@ -182,12 +182,16 @@ async fn assert_chat_quiet(socket: &mut JsonSocket) { ); } -/// The `(role, content)` pairs of one captured request's message list. +/// The `(role, content)` pairs of one captured request's message list, +/// without its leading system messages: these gates pin the conversation, +/// so they read the same whether or not the chat agent opens with a +/// system message. fn role_content_pairs(request: &serde_json::Value) -> Vec<(String, String)> { request["messages"] .as_array() .expect("a captured request includes a messages array") .iter() + .skip_while(|message| message["role"] == "system") .map(|message| { ( message["role"] diff --git a/crates/workshop/server/tests/it/chat_gate/canonical_sequence.rs b/crates/workshop/server/tests/it/chat_gate/canonical_sequence.rs index 833702ab1..c9b1397c4 100644 --- a/crates/workshop/server/tests/it/chat_gate/canonical_sequence.rs +++ b/crates/workshop/server/tests/it/chat_gate/canonical_sequence.rs @@ -114,6 +114,36 @@ async fn gate_history_accumulates_across_three_turns_byte_exact() { socket.close().await; } +/// GATE 13 - file tools. The chat offers the extra Packs, so its rounds +/// advertise the `std` file tools the model reads the language guide with. +#[tokio::test] +async fn gate_a_conversation_round_advertises_the_file_tools() { + let server = spawn_chat_server(&["test-model"]).await; + let mut socket = connect_chat(&server.ws_base).await; + let _session = launch_chat(&mut socket).await; + + let token = next_wait_token(&mut socket).await; + answer(&mut socket, &token, "ping").await; + let turn = collect_turn(&mut socket).await; + assert_eq!(delta_text(&turn), "echo:ping"); + { + let requests = server.captured.lock().expect("the capture lock is healthy"); + let names: Vec<&str> = requests[0]["tools"] + .as_array() + .expect("the conversation round advertises tools") + .iter() + .filter_map(|tool| tool["function"]["name"].as_str()) + .collect(); + for wire in ["Read", "ListDir", "Grep"] { + assert!( + names.contains(&wire), + "the conversation round advertises {wire}, got {names:?}" + ); + } + } + socket.close().await; +} + /// GATE 12 - web tools. The chat's `## Conversation` section offers the /// `web` Pack, so the rounds its model loop sends advertise the /// Pack's fetch and search tools; an offer in the H1 would not reach diff --git a/crates/workshop/server/tests/it/chat_gate/compaction.rs b/crates/workshop/server/tests/it/chat_gate/compaction.rs index cb1ee8ac8..91d475094 100644 --- a/crates/workshop/server/tests/it/chat_gate/compaction.rs +++ b/crates/workshop/server/tests/it/chat_gate/compaction.rs @@ -4,7 +4,8 @@ use super::*; /// The most messages the compacting mock admits in one request; a longer -/// one is refused as a context overflow. +/// one is refused as a context overflow. System messages do not count, so +/// the gate is the same whether or not the chat agent opens with one. const MESSAGE_LIMIT: usize = 4; /// The opening of the chat compactor's summary request. @@ -53,7 +54,11 @@ fn compacting_completions(captured: &CapturedRequests, body: &str) -> Response { let messages = request["messages"] .as_array() .expect("the request includes a messages array"); - if messages.len() > MESSAGE_LIMIT { + let counted = messages + .iter() + .filter(|message| message["role"] != "system") + .count(); + if counted > MESSAGE_LIMIT { let refusal = json!({ "error": { "message": "This request exceeds the model's maximum context length.", "code": "context_length_exceeded", diff --git a/crates/workshop/server/tests/it/main.rs b/crates/workshop/server/tests/it/main.rs index b6de8576c..7ac67a355 100644 --- a/crates/workshop/server/tests/it/main.rs +++ b/crates/workshop/server/tests/it/main.rs @@ -1,9 +1,9 @@ //! Workshop server integration tests, one module per behavior area: the //! `/ws` socket and its heartbeat-driven frames, the `/agents/ws` socket //! and the built-in chat agent's gate, the realtime relay, the -//! heartbeat loop, boot composition, the user-state bucket, the -//! workspace file across a graceful shutdown, and the workspace save -//! deadline through the full router. +//! heartbeat loop, boot composition, the user-state bucket, the std file +//! tools over a real directory, the workspace file across a graceful +//! shutdown, and the workspace save deadline through the full router. #[path = "../common/mod.rs"] mod common; @@ -14,6 +14,7 @@ mod chat_gate; mod heartbeat; mod heartbeat_loop; mod realtime_relay; +mod std_real_directory; mod user_state; mod workshop_socket; mod workspace_shutdown; diff --git a/crates/workshop/server/tests/it/std_real_directory.rs b/crates/workshop/server/tests/it/std_real_directory.rs new file mode 100644 index 000000000..049be6be6 --- /dev/null +++ b/crates/workshop/server/tests/it/std_real_directory.rs @@ -0,0 +1,167 @@ +//! The `std` file tools over a real directory, mounted with +//! `RealBackend::rooted`: the acceptance flow a coding agent follows. List +//! the directory, find a word with Grep, read a window of the file, and read +//! the next window from the offset the first window's trailer names. +//! +//! This lives here and not in `plugin-std` because the crate needs the +//! real backend, and a Plugin crate imports only `runvil-plugin`. + +use std::future::Future; +use std::pin::pin; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::task::{Context, Poll, Waker}; + +use runvil::effect::{ToolCallOrigin, ToolCaller}; +use runvil::vfs::{Access, Origin, RealBackend, VfsRef}; +use runvil_plugin::{ + HostServices, OutputTrust, Pack, PackId, PackRegistry, ToolContext, ToolId, ToolOutput, +}; +use serde_json::{Value, json}; + +/// The file the flow runs over. +const FILE: &str = "big.txt"; + +/// How many lines the file has. +const LINES: usize = 5000; + +/// The line that holds the word Grep finds. +const NEEDLE_LINE: usize = 3000; + +/// The word only that line holds. +const NEEDLE: &str = "needleword"; + +/// The first line the second window starts at, which is the line after the +/// last one a 100,000-byte window of 48-byte numbered lines holds. +const SECOND_OFFSET: usize = 2084; + +/// Counts the directories this process has made, so two tests never share +/// one. +static DIRECTORIES: AtomicUsize = AtomicUsize::new(0); + +/// A directory named from the test, the process id, and a counter, holding +/// the file the flow reads. +// clippy.toml's allow-expect-in-tests covers #[test] functions only, not +// their helpers; failing the test by panicking with the invariant named +// is exactly what this is for. +#[expect( + clippy::expect_used, + reason = "test helpers fail by panicking with the invariant named" +)] +fn seeded_directory() -> tempfile::TempDir { + let name = format!( + "std-real-directory-{}-{}-", + std::process::id(), + DIRECTORIES.fetch_add(1, Ordering::Relaxed) + ); + let directory = tempfile::Builder::new() + .prefix(&name) + .tempdir() + .expect("a temporary directory is made"); + let mut text = String::new(); + for number in 1..=LINES { + if number == NEEDLE_LINE { + text.push_str(NEEDLE); + text.push_str(&"x".repeat(40 - NEEDLE.len())); + } else { + text.push_str(&"x".repeat(40)); + } + text.push('\n'); + } + std::fs::write(directory.path().join(FILE), text).expect("the file is written"); + directory +} + +/// The Pack `plugin_std::install` inserts, built under the Pack id `std`. +#[expect( + clippy::expect_used, + reason = "test helpers fail by panicking with the invariant named" +)] +fn std_pack() -> std::sync::Arc { + let mut packs = PackRegistry::new(); + plugin_std::install(&mut packs, Value::Null).expect("a fresh registry takes the Pack"); + let (info, _id, config) = packs + .into_entries() + .find(|(info, _, _)| info.fqn == "runvil/std") + .expect("install inserts the std Pack"); + let name = PackId::parse("std").expect("a one-segment Pack id parses"); + (info.construct)(&name, config, &HostServices::new()).expect("the Pack is built") +} + +/// Drives a future to its output, polling again after each `Pending`. +fn block_on(future: F) -> F::Output { + let mut future = pin!(future); + let mut cx = Context::from_waker(Waker::noop()); + loop { + if let Poll::Ready(output) = future.as_mut().poll(&mut cx) { + return output; + } + } +} + +/// Calls the tool `segment` of `pack` with `args` over `access`. +#[expect( + clippy::expect_used, + reason = "test helpers fail by panicking with the invariant named" +)] +fn call(pack: &dyn Pack, access: &Access, segment: &str, args: Value) -> ToolOutput { + let tool = ToolId::parse(&format!("std/{segment}")).expect("a two-segment tool id parses"); + let origin = ToolCallOrigin::new("std-real-directory", "Test", ToolCaller::Script); + let services = HostServices::new(); + let context = ToolContext::new(&tool, access, &origin, &services); + block_on(pack.call(context, args)).expect("the call succeeds") +} + +#[test] +fn list_find_read_a_window_and_read_the_next_one_by_the_trailer_offset() { + let directory = seeded_directory(); + let backend = RealBackend::rooted(directory.path()).expect("the directory mounts"); + let vfs = VfsRef::builder().mount("/", backend).build(); + let access = vfs + .acquire(Origin::new("std real directory")) + .expect("the run acquires the filesystem"); + let pack = std_pack(); + + let listing = call(&*pack, &access, "dir", json!({ "target_directory": "/" })); + assert_eq!(listing.trust(), OutputTrust::Untrusted); + assert_eq!(listing.text(), format!("/\n - {FILE}")); + + let found = call(&*pack, &access, "grep", json!({ "pattern": NEEDLE })); + assert_eq!(found.trust(), OutputTrust::Untrusted); + let expected_line = format!(" {NEEDLE_LINE}:{NEEDLE}{}", "x".repeat(40 - NEEDLE.len())); + assert_eq!(found.text(), format!("/{FILE}\n{expected_line}")); + + let path = format!("/{FILE}"); + let first = call(&*pack, &access, "read", json!({ "path": path })); + assert_eq!(first.trust(), OutputTrust::Untrusted); + assert!( + first + .text() + .starts_with(&format!(" 1|{}\n", "x".repeat(40))), + "the first window starts at line 1" + ); + let trailer = first + .text() + .lines() + .last() + .expect("a first window has a last line"); + assert!( + trailer.contains(&format!("Use offset={SECOND_OFFSET} to continue.")), + "{trailer}" + ); + + let second = call( + &*pack, + &access, + "read", + json!({ "path": path, "offset": SECOND_OFFSET }), + ); + assert!( + second.text().starts_with(&format!(" {SECOND_OFFSET}|")), + "the second window starts at the offset the trailer named: {}", + &second.text()[..20] + ); + assert!( + second.text().contains(&format!(" {NEEDLE_LINE}|{NEEDLE}")), + "the second window holds the line Grep found" + ); +} diff --git a/crates/workshop/ui/src/parts/run/run-rows.ts b/crates/workshop/ui/src/parts/run/run-rows.ts index eba98aeb7..0c4c05f38 100644 --- a/crates/workshop/ui/src/parts/run/run-rows.ts +++ b/crates/workshop/ui/src/parts/run/run-rows.ts @@ -182,10 +182,14 @@ export function renderContractRows( } for (const model of contract.models) { - const details = - model.minContext === null - ? model.keywords.join(", ") - : `${model.keywords.join(", ")} (min context ${model.minContext})`; + const parts: string[] = []; + if (model.minContext !== null) { + parts.push(`min context ${model.minContext}`); + } + if (model.thinking !== null) { + parts.push(model.thinking ? "thinking" : "no thinking"); + } + const details = parts.join(", "); const modelRow = row(model.label, [readOnly(details)]); modelRow.classList.add("ws-run-panel__row--model"); rows.appendChild(modelRow); diff --git a/crates/workshop/ui/src/services/run-api.ts b/crates/workshop/ui/src/services/run-api.ts index 3dad76326..814fa10d4 100644 --- a/crates/workshop/ui/src/services/run-api.ts +++ b/crates/workshop/ui/src/services/run-api.ts @@ -42,13 +42,14 @@ export interface RunContractArgs { readonly fields: readonly RunContractArg[]; } -/** One declared model role. */ +/** The requirements a prompt declares for one model role. */ export interface RunContractModel { + /** The role name. */ readonly label: string; - readonly keywords: readonly string[]; /** The minimum context window in tokens; null when absent. */ readonly minContext: number | null; - readonly description: string | null; + /** The thinking requirement; null when absent. */ + readonly thinking: boolean | null; } /** The Run-window contract: everything the panel renders. */ @@ -146,25 +147,20 @@ function parseModel(value: unknown): RunContractModel | null { if (!isRecord(value)) { return null; } - const { label, keywords, min_context } = value; + const { label, min_context, thinking } = value; if (typeof label !== "string") { return null; } - if (!Array.isArray(keywords) || !keywords.every((keyword) => typeof keyword === "string")) { - return null; - } if (min_context !== null && typeof min_context !== "number") { return null; } - const description = value.description; - if (description !== null && typeof description !== "string") { + if (thinking !== null && typeof thinking !== "boolean") { return null; } return { label, - keywords, minContext: min_context ?? null, - description: description ?? null, + thinking: thinking ?? null, }; } diff --git a/crates/workshop/ui/test/run-api.mjs b/crates/workshop/ui/test/run-api.mjs index f5072c1ac..3abe53d94 100644 --- a/crates/workshop/ui/test/run-api.mjs +++ b/crates/workshop/ui/test/run-api.mjs @@ -48,7 +48,7 @@ const CONTRACT = { ], }, models: [ - { label: "main", keywords: ["thinking", "frontier"], min_context: 32000, description: null }, + { label: "strong", min_context: 32000, thinking: true }, ], }; @@ -122,9 +122,9 @@ const json = (body, status = 200) => ({ check( "model roles narrow", contract.models.length === 1 && - contract.models[0].label === "main" && - contract.models[0].keywords.join(",") === "thinking,frontier" && - contract.models[0].minContext === 32000, + contract.models[0].label === "strong" && + contract.models[0].minContext === 32000 && + contract.models[0].thinking === true, ); } @@ -166,6 +166,44 @@ const json = (body, status = 200) => ({ ); } +// --- Model requirements: null forms narrow to null, a bad shape is refused --- + +{ + scriptFetch(() => + json({ + ...CONTRACT, + models: [{ label: "fast", min_context: null, thinking: null }], + }), + ); + const contract = await fetchPromptContract("p.md", "text"); + check( + "a role with no requirements narrows both to null", + contract.models.length === 1 && + contract.models[0].label === "fast" && + contract.models[0].minContext === null && + contract.models[0].thinking === null, + ); +} + +{ + scriptFetch(() => + json({ + ...CONTRACT, + models: [{ label: "strong", min_context: null, thinking: "yes" }], + }), + ); + let caught = null; + try { + await fetchPromptContract("p.md", "text"); + } catch (error) { + caught = error; + } + check( + "a model entry whose thinking is a string is a shape failure", + caught !== null && caught.code === "unexpected_shape", + ); +} + // --- Narrowing rejects unknown variants ------------------------------------- { diff --git a/crates/workshop/ui/test/run-panel.mjs b/crates/workshop/ui/test/run-panel.mjs index e9239ab2f..7e52c3c97 100644 --- a/crates/workshop/ui/test/run-panel.mjs +++ b/crates/workshop/ui/test/run-panel.mjs @@ -134,7 +134,9 @@ function contractFor(name) { ], }, models: [ - { label: "main", keywords: ["thinking", "frontier"], min_context: 32000, description: null }, + { label: "strong", min_context: 32000, thinking: true }, + { label: "small", min_context: null, thinking: false }, + { label: "fast", min_context: null, thinking: null }, ], }; } @@ -436,12 +438,18 @@ check( "the ready panel renders no tool row", rows.length > 0 && filledEl.querySelector(".ws-run-panel__row--tool") === null, ); -const modelRow = rowText("main"); +const modelRow = rowText("strong"); check( - "a model role renders keywords and min_context read-only", - modelRow?.textContent.includes("thinking") === true && - modelRow?.textContent.includes("frontier") === true && - modelRow?.textContent.includes("32000"), + "a model role renders min_context and thinking read-only", + modelRow?.textContent.includes("min context 32000, thinking") === true, +); +check( + "a role that requires no thinking reads no thinking", + rowText("small")?.textContent.includes("no thinking") === true, +); +check( + "a role with no requirements shows only its label", + rowText("fast")?.textContent === "fast", ); const iterationsField = rowText("max_tool_iterations")?.querySelector('input[type="number"]'); check("max_tool_iterations prefills from the contract", iterationsField?.value === "12"); diff --git a/prompts/analyst-example.md b/prompts/analyst-example.md index 0750bc048..c07bd1207 100644 --- a/prompts/analyst-example.md +++ b/prompts/analyst-example.md @@ -3,21 +3,20 @@ name: analyst_example description: Demonstrate frontmatter model roles and models.use for careful model selection. runvil: 0 models: - analyst: + strong: min_context: 40000 - description: A model suited for careful analysis --- # Analyst Example --- -Demonstrates prompt-local model selection. The frontmatter `models:` key declares the `analyst` role, filled from the Host's current model at prepare. A section that calls `models.use` runs every completion under that role's bound model; a section that omits `models.use` inherits the prompt-wide `models.default` model when one is declared. +Demonstrates model selection by role. The frontmatter `models:` key sets a context minimum for the `strong` role, and the Host decides which model fills that role. A prompt whose `strong` model is too small is refused before it runs. A section that calls `models.use("strong")` runs its completions on that role's model, and a section that omits `models.use` runs on the `default` role. ## Analyze ```lua -models.use("analyst") +models.use("strong") ``` Analyze the following input carefully and return a short factual summary with no preamble: diff --git a/prompts/greet.md b/prompts/greet.md index 8940ac4c7..b3e73761c 100644 --- a/prompts/greet.md +++ b/prompts/greet.md @@ -2,16 +2,10 @@ name: greet description: Greet the named input using a Lua-computed value runvil: 0 -models: - writer: {} --- # Greet -```lua -models.default("writer") -``` - Computes a greeting from the input in Lua, substitutes it into the prose, and has the model echo it. diff --git a/prompts/hello.md b/prompts/hello.md index f67c547c5..5ecc5c6ac 100644 --- a/prompts/hello.md +++ b/prompts/hello.md @@ -2,16 +2,10 @@ name: hello description: Say hello runvil: 0 -models: - writer: {} --- # Hello World -```lua -models.default("writer") -``` - A minimal test prompt. ## Greet diff --git a/prompts/research-person.md b/prompts/research-person.md index 8b4901929..8b52f1480 100644 --- a/prompts/research-person.md +++ b/prompts/research-person.md @@ -5,16 +5,10 @@ runvil: 0 max_tool_iterations: 20 uses: - web -models: - researcher: {} --- # Research a Person -```lua -models.default("researcher") -``` - ## Research ```lua diff --git a/vibe/2026-10-31-1-model-only-call-flag.md b/vibe/2026-10-31-1-model-only-call-flag.md new file mode 100644 index 000000000..f7dc77cd6 --- /dev/null +++ b/vibe/2026-10-31-1-model-only-call-flag.md @@ -0,0 +1,206 @@ +--- +name: model_only_call flag +overview: "`ToolInfo` gains a `model_only_call` flag, so a Pack can keep a tool in the catalog where Lua names and offers it, while a script's `tools.call` refuses it. The model still calls it, because the arguments a render shaped belong to the model that receives them." +todos: + - id: flag + content: "Add `model_only_call` to `ToolInfo`: field, builder, `false` in `new`, serde default, and the module doc exception" + status: completed + - id: binding + content: Carry the flag on `ToolBinding` through `from_info`, and copy it in `scope::bind` so a render cannot clear it + status: completed + - id: refuse + content: Refuse a script `tools.call` in `tool_route` with a new `Error::ModelOnlyToolCall`, classified as `RunErrorKind::Tool` + status: completed + - id: lua-surface + content: Expose `model_only_call` on the Lua tool object, so a prompt reads it before offering the tool + status: completed + - id: tests + content: Add the refusal, offerable, and model-still-calls tests, and update the exact-JSON tests the always-emitted key changes + status: completed + - id: gates + content: Run the crate and workspace suites, clippy, fmt, rustdoc, build-xtask, the wording scan, and `xtask api --bless` for the facade listing + status: completed +isProject: false +--- + +# The model_only_call flag + + + +## Product Requirements + +A Pack may render a tool differently per model: different parameter names, a different schema, a different description. Lua has no way to read a schema, so no script can know what arguments such a tool takes. The tool must stay in the catalog anyway, because `tools.offer("std/replace")` has to name it. This plan adds the one flag that resolves the tension. + +- Problem and users: + - A model-shaped tool's arguments belong to the model the render served. A script that calls it writes JSON from a guess. + - The users are Plugin authors, who need to declare the tool un-callable from Lua, and prompt authors, who need to read that before offering it. +- Goals: + - Three facts stay independent for one tool: it is in the catalog, Lua may offer it, and Lua may or may not call it. + - The flag lives on the catalog entry, where the Pack sets it, and a render cannot clear it. + - The refusal is a normal Lua error the author can catch, not a run failure. +- Non-goals: + - No tool becomes model-shaped here. Nothing in `plugin-std` sets the flag, and no definition gains a `render`. + - No change to `Pack::render`, `RenderContext`, `PackInfo`, or the merge rules. + - No change to the discovery tools or the Dynamic Set, which resolve as they do today. + - No new Lua verb. The flag is a read-only field on a tool object already reachable through `tools.get`. +- Success criteria: + - A script `tools.call` of a model-only tool fails with a typed error naming the id, and `pcall` catches it. + - The same tool offered to the model is advertised and its call answers, which proves the flag gates the script alone. + - `tools.offer` accepts the tool alone and with its Pack, so the flag never hides anything. + - A render returning a catalog tool with the flag cleared still binds it model-only. +- Constraints: + - The four flags on `ToolInfo` are independent questions about one tool, so they stay bools. + - The key is always serialized, as `survives_stop` and `describes_tools` are, so a recorded run shows it. + - Engine crate docs say "the caller", never the Host. +- Open questions: None. + +## Functional Specification + +A tool object in Lua gains a fifth field. A script that calls a model-only tool by id or by object gets an error it can catch. A model round is unaffected. + +- Actors and workflows: + - A Plugin author marks a tool with `ToolInfo::model_only_call(true)` in `Pack::tools`. + - A prompt author reads `tools.get("std/replace").model_only_call` and decides whether to offer it. + - A script calls `tools.call(id, args)` or `tools.call(object, args)`: both reach the same lookup, so both are refused. + - A model round calls the tool by its wire name; the frozen sets resolve it and the call answers. +- Inputs and outputs: + - The tool object's fields are `id`, `description`, `parameters`, `pack`, and `model_only_call`, all read-only, and no others. + - The refusal's message is `tool "tools/replace" (tools/replace) is model-only: a render shapes its arguments for the model that receives them, so a script cannot call it; offer it with tools.offer and let the model call it`. + - A Lua error table carries the kind tag `model_only_tool` and the `name` field. +- States and validation: + - `ToolInfo::new` leaves the flag false, so every existing tool keeps its behavior. + - A serialized `ToolInfo` without the key reads `false`, so a run log written before this change still loads. + - `bind` copies the catalog's value onto a rendered tool with a catalog id, so a render that clears the flag changes nothing. +- Errors and recovery: + - The refusal is a Lua error at the `tools.call` boundary, like an unbound id today. + - An uncaught refusal ends the section as a tool failure under `RunErrorKind::Tool`. +- Security and privacy behavior: + - No new data reaches the model or a script. The flag is the Pack's own declaration about its own tool. + - The refusal names only the id and a name the caller already passed. +- Acceptance criteria: + - `--workspace` and the workshop suites pass, with the crate suites for `runvil-types`, `runvil-lua`, and `runvil-engine`. + - `cargo xtask api --check` passes, and the facade gains exactly the field and the builder. + + + + +## Technical Design + +The change is one field carried through four layers that already carry the same three flags: `ToolInfo`, `ToolBinding`, the frozen-set merge, and the Lua tool object. The only new branch is in the script half of one lookup function. + +- Architecture: + - `ToolInfo` in `crates/runvil-internal/types/src/tools/info.rs` gains `pub model_only_call: bool` with `#[serde(default)]`, a builder, and `false` in `new`. The crate doc's claim that every catalog tool is callable from Lua now names the exception. + - `ToolBinding` in `crates/runvil-internal/lua/src/bindings.rs` carries the flag and an accessor, copied in `from_info`. + - `bind` in `crates/runvil-internal/engine/src/execute/scope.rs` copies the catalog's value onto a rendered tool, beside `structured_output`, `describes_tools`, and `survives_stop`. + - `LuaToolHandle` in `crates/runvil-internal/lua/src/tools/userdata.rs` gains the field and a read-only getter. +- Modules and interfaces: + - `Error::ModelOnlyToolCall { name, id }` in `crates/runvil-internal/engine/src/error.rs`, beside `UnboundToolCall`. It joins the `None` arm of `originating_error`. + - `RunErrorKind::Tool` classifies it, in `crates/runvil-internal/engine/src/execute/error.rs`. + - `runvil_lua::ErrorKind::ModelOnlyTool` renders it to Lua with the tag `model_only_tool`, joining `ALL`, `tag`, `from_tag`, and the two exhaustive matches in `crates/runvil-internal/lua/src/vm/run.rs` and `crates/runvil-internal/engine/src/error/convert.rs`. + - `tool_route` in `crates/runvil-internal/engine/src/execute/scheduler/tool_call.rs` refuses in the script branch, after the catalog lookup, so the bare id and a tool object both pass through one check. +- File and public API changes: + - `crates/runvil/public-api.txt` gains `ToolInfo::model_only_call` as a field and as a builder. `ToolInfo` is on the facade, so the listing is regenerated with `cargo +nightly-2026-09-05 xtask api --bless`. + - The Lua error vocabulary gains one tag, which a prompt may branch on. +- Data, persistence, failure, security, and privacy constraints: + - The key is always emitted, so a recorded `ToolInfo` shows the flag rather than leaving a reader to infer it. Three tests that pin exact JSON lines are updated, and the test that pinned a pre-`wire_name` line now asserts the absent flags read false instead. + - No new dependency, no unsafe code, and no filesystem access. + + + + +## Testing Plan + +- Unit: + - **Types:** the builder sets the flag and serde round-trips it; a line without the key reads false; the canonical form holds the key in sorted position. + - **Binding:** `from_info` copies the flag, and a new info leaves it false. + - **Lua surface:** the tool object reads `model_only_call` for a marked tool, and the field-set test still refuses a sixth field. +- Integration and end-to-end: + - **Refusal** in `failures-script-tools.rs`: the id form fails with `ModelOnlyToolCall` naming the id, the object form fails the same way, and `pcall` catches it with the run continuing. + - **Offer still works:** `tools.offer` accepts the model-only tool alone and through its Pack. + - **Model path** in `offering.rs`: a section offers the tool, a scripted round calls it by wire name, and the `ToolCall` effect names the right id and alias. +- Regression, security, and performance: + - The exact-JSON assertions in `types/src/tools/tests.rs` and `execute/run/effect-tests.rs` pin the serialized shape, so a later change to the key fails rather than drifting. +- Exit criteria, every command below passes: + - `cargo nextest run --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-features`, then `cargo nextest run --locked -p workshop -p workshop-server -p workshop-server-api`. + - Both clippy runs with `CARGO_BUILD_WARNINGS=deny`, and `cargo fmt --all --check`. + - `RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --all-features --exclude workshop --exclude workshop-server --exclude workshop-server-api`, and the facade's own doc gate. + - `cargo test -p build-xtask`, `node --test crates/workshop/ui/test/docs-claims.mjs`, and `cargo +nightly-2026-09-05 xtask api --check`. + + + + +## Decision Record + +- Decisions: + - **A flag on the catalog entry, not a model-only tool minted inside a render.** Rationale: Lua must be able to name the tool to offer it, and a tool that exists only in a render's answer cannot be named at all. The catalog entry stays and the call is refused. Product owner: "Needs to be a flag on the Lua facing tool that says that the Lua is not allowed to call it." + - **The render may not clear it.** Rationale: the catalog is the authority on what a Pack offers, and the other three flags already work this way. A render that could clear it would let a Pack grant itself script-callable status its own catalog denied. + - **A typed error with its own Lua kind.** Rationale: an author needs to branch on "this is model-only" differently from "there is no such tool", and a distinct tag keeps the two apart in a caught error. + - **Four bools, kept as bools.** Rationale: each answers a different question about one tool, so no state machine or enum expresses them. Clippy's `struct_excessive_bools` is expected with that reason rather than silenced. + - **The key is always serialized.** Rationale: a recorded run should show the flag, and the cost is three exact-line tests. +- Rejected alternatives: + - Minting a model-only tool inside `Pack::render`. Rejected: Lua cannot name it, so a prompt could never offer it individually. + - One `callable: Callers` enum instead of a bool. Rejected: it reads as a state machine over exclusive states, and these flags are independent. + - Failing the run instead of raising a catchable Lua error. Rejected: an author may want to offer the tool and let the model call it, which is the whole point. +- Assumptions, risks, and notes: + - Nothing uses the flag yet. The first real consumer is `std/replace`, which is a separate change. + - `ToolInfo` is on the facade, so this is a public-surface addition, though a purely additive one. + +### Deferred and Out of Scope + +- Deferred: `std/replace` marked model-only with a `render` that shapes its schema per model. +- Out of scope: any change to `Pack::render`, `RenderContext`, or `PackInfo`, and any change under `crates/plugin-std`. + + + + +## Project Survey + +- Status: complete +- Build command: `cargo build -p runvil` (the workspace default members are `runvil`, `runvil-plugin`, `harness`, and `harness-gateway-client`). +- Focused test command pattern: `cargo nextest run --locked -p --all-features `; drop `--all-features` for `workshop`, `workshop-server`, and `workshop-server-api`. +- Component test command pattern: `cargo nextest run --locked -p --all-features`; structural checks: `cargo test -p build-xtask`. +- Full-suite test command: `cargo nextest run --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-features`, then `cargo nextest run --locked -p workshop -p workshop-server -p workshop-server-api`. +- Linter command: `CARGO_BUILD_WARNINGS=deny cargo clippy --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-targets --all-features` and the workshop run beside it; wording scan: `node --test crates/workshop/ui/test/docs-claims.mjs`. +- Formatter check command: `cargo fmt --all --check`. +- Docs command: `RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --all-features --exclude workshop --exclude workshop-server --exclude workshop-server-api`, plus `RUSTDOCFLAGS="-D warnings" cargo doc -p runvil --no-deps`; facade surface: `cargo +nightly-2026-09-05 xtask api --check` with `CARGO_BUILD_WARNINGS` unset. +- Test placement and naming conventions: unit tests sit beside their module in a `-tests.rs` sibling wired by `#[cfg(test)] #[path = "-tests.rs"] mod tests;`. Engine integration tests live under `execute/tests/`, one module per topic. +- Directory map: `crates/runvil-internal/` holds the Engine's private crates (`types`, `parser`, `lua`, `vfs`, `model-client`, `engine`) behind the `runvil` facade. `crates/runvil-plugin` is the Plugin API. +- Component boundaries: Engine crates depend on no gateway, workshop, or Harness crate, and `cargo test -p build-xtask` enforces it. `runvil-plugin` re-exports the `runvil-types` and `runvil-vfs` names a Plugin author needs, so a Plugin crate names no other runvil crate. +- Conventions summary: Rust 2024, workspace version 0.3.0, BSL-1.0; workspace lints inherited with pedantic denied and `unwrap_used`/`expect_used` denied outside tests; every Rust file at most 500 lines, enforced by each crate's `build.rs`; capitalized defined terms checked by `docs-claims.mjs`; model-readable error messages naming required versus actual; no doctests anywhere. + + + + +## Execution Instructions + +Each step landed as one commit-sized change, and every step's checks passed before the next began. + + +Add `model_only_call` to `ToolInfo` in `crates/runvil-internal/types/src/tools/info.rs`: the field with `#[serde(default)]`, the builder, `false` in `new`, and the crate doc's exception. Expect `clippy::struct_excessive_bools` at the fourth bool and record it with `#[expect]` and the reason that the flags are independent. + + + +Carry the flag on `ToolBinding` in `crates/runvil-internal/lua/src/bindings.rs` through `from_info`, with an accessor, and copy it in `bind` in `crates/runvil-internal/engine/src/execute/scope.rs` so a render cannot clear the catalog's declaration. + + + +Add `Error::ModelOnlyToolCall { name, id }` in `crates/runvil-internal/engine/src/error.rs`, classify it `RunErrorKind::Tool` in `execute/error.rs`, and add `runvil_lua::ErrorKind::ModelOnlyTool` with its tag `model_only_tool` across `error-value.rs`, `vm/run.rs`, and `error/convert.rs`. + + + +Refuse in `tool_route`'s script branch after the catalog lookup, so the bare id and a tool object both pass through the one check. Use a `let ... else` on the lookup rather than `map(..).unwrap_or_else(..)`, which clippy refuses. + + + +Expose the flag on the Lua tool object in `crates/runvil-internal/lua/src/tools/userdata.rs`, and update the module doc and the field-set test that pins exactly five fields. + + + +Add the tests: the two refusal forms, the `pcall` catch, the two offer forms, the binding copy, the serde round trip, and a model round that advertises and calls the tool. Update the three exact-JSON assertions the always-emitted key changes. + + + +Run every gate in the Testing Plan, blessing the facade listing with `cargo +nightly-2026-09-05 xtask api --bless` and confirming `--check` afterwards. + + + diff --git a/vibe/2026-10-31-2-built-in-model-roles.md b/vibe/2026-10-31-2-built-in-model-roles.md new file mode 100644 index 000000000..cf0be0c84 --- /dev/null +++ b/vibe/2026-10-31-2-built-in-model-roles.md @@ -0,0 +1,685 @@ +--- +name: Built-in model roles +overview: "Every prompt gets exactly five roles (default, strong, small, fast, creative), no custom roles, and models: front matter only for requirements (min_context, thinking). The Host passes a plain role-to-model map to prepare; a role with no entry uses default, and a non-empty map without default is refused. The default role is the baseline, models.current() reads the chain's model, models.use() switches it for the rest of the chain, children inherit a copy, and a 3-arg fanout sets the arms' model. Built with fewer types than exist today. Three commits." +todos: + - id: types-model-role + content: "runvil-types models.rs: ModelRole enum (Default, Strong, Small, Fast, Creative) with ALL, as_str() and serde Deserialize (kebab-case); unit tests" + status: completed + - id: parser-requirements + content: "Parser: rename ModelRole struct to RoleRequirements { min_context, thinking: Option }; delete ModelKeyword, keywords:, description:, ModelRoles, ModelRolesIter, keyword_name; add Frontmatter::role_requirements(role); map visitor generic over key type (args keep alias grammar); update workshop ModelDto and the UI's run-api.ts and run-rows.ts; rewrite contract tests" + status: completed + - id: modelset-and-lua + content: "model-client: ModelSet immutable (delete default alias API, ModelView, ModelSetError, lock helper, capabilities); Lua: Arc; selection is Arc>> passed into injection and initialized to the default binding; models.use(label or handle, opts?); models.current; delete models.default and resolve_model_binding (SectionVm::current_model); handle loses capabilities; role errors" + status: completed + - id: engine-prepare + content: "Engine: prepare(prompt, BTreeMap) checks declared requirements against the filling model, refuses a non-empty map without default (RequirementCheck::Unbound), and builds the ModelSet on RunContext; delete ModelBindings, bindings.rs, model_bindings(), bind_role_for_test, Scenario::bind_roles; RequirementCheck {ContextMinimum, Thinking, Unbound}; UnmetRequirement.role: ModelRole; RunState holds Arc" + status: completed + - id: callers-and-fixtures + content: "One pass per file: prepare callers (harness prepare_from_snapshot, scenarios, example, bench, tests) and every prompt and fixture with custom labels, keywords or models.default (README, prompts, workshop chat.md, roughly 30 fixture files); regenerate public-api.txt; first design doc update" + status: completed + - id: gate-commit-1 + content: "Commit 1 gate: cargo test and clippy for types, parser, model-client, lua, engine, harness, facade, workshop server; UI build; stale-name and wording sweeps; check that model-call events record the model id actually used" + status: completed + - id: chain-selection + content: "Commit 2: Chain owns the selection Arc; ChainStart carries the starting value (default binding for root chains, parent's value for call and spawn, taken at dispatch); end_h1 hands H1's value to the root walk; section setup passes the chain's Arc into injection; tests; design doc update" + status: completed + - id: fanout-shim + content: "Commit 3: 3-arg tasks.fanout(model, worker, collection) entirely in __impl_fanout.lua using privileged nil-safe selection get/set captured at install: switch, spawn every arm, restore before the first join and on the spawn-failure path; tests; design doc update" + status: completed + - id: final-gate + content: "Gates for commits 2 and 3: tests, clippy, public-api check, stale-name and wording sweeps" + status: completed +isProject: false +--- + +# Built-in model roles and chain-level current model + + + +## Product Requirements + +Prompt authors declare model roles in `models:` front matter and pick one with `models.default`, although the Host binds every declared role to a single model. This change gives every prompt five fixed roles, makes `models:` optional and limited to requirements, and lets the Host map roles to models. The current model becomes state of the chain, inherited by its children. It ships with a smaller API than exists today and with no compatibility paths. + +- Problem and users: + - Prompt authors must write a `models:` block naming a role label and a `models.default(label)` call before the first model call, then the Host binds every declared role to its one current model (`crates/runvil-internal/parser/src/contract/models.rs`, `crates/runvil-internal/engine/src/execute/fill.rs`, `README.md`). + - Host integrators (the harness, the Workshop server, embedders of `RunContext`) pass one optional model to `prepare` and cannot express per-role models (`crates/runvil-internal/engine/src/execute/config.rs`). + - Authors cannot carry a model choice across sections or into fanout arms: the selection lives on the section VM and each new VM starts without one, so arms and later sections fall back to the prompt-wide default (`crates/runvil-internal/lua/src/vm.rs`, `crates/runvil-internal/lua/src/models.rs`). +- Goals: + - A prompt with no `models:` block runs on the Host's default model. + - Five fixed roles (`default`, `strong`, `small`, `fast`, `creative`) that a prompt may attach requirements to. + - A Host maps roles to models with one plain map, and a role without an entry uses `default`. + - `models.current()` and `models.use(...)` act on a chain-level current model that `call` children, spawned tasks and fanout arms inherit, and a 3-arg `tasks.fanout` sets the arms' model. + - The same observable results with fewer types and functions than exist today. +- Non-goals: + - Custom prompt-declared roles. + - Any compatibility path, alias, stub, migration error, or note about previous behavior. + - A role for compaction. + - Per-role model selection in production Hosts. +- Success criteria: + - A prompt file with no `models:` block and no `models.default` call runs unchanged against a Host that supplies one model. + - Every deleted name listed in the sweep (Testing Plan) is absent from living code and docs. + - Every gate in Execution Instructions passes. +- Constraints: + - Code, doc comments, error messages, test names, README text and the design document describe present behavior only. None says "replaces", "formerly", "previously", "no longer", "legacy", "removed" or "migrated", and no test asserts that a deleted name is gone. + - Prompts, fixtures and docs are rewritten in place, with no "was X, now Y" annotations. + - Dated logs under `vibe/` and the generated `.vibelight/` directory are not edited. + - Each step commit leaves the workspace building with its tests passing. + - A run's result does not depend on when the scheduler admits a queued task: inherited model state is copied when a call or spawn is dispatched. +- Open questions: None + +## Functional Specification + +The author-facing surface is five fixed roles, optional requirements in front matter, and the Lua calls `models.current`, `models.use`, `models.get` and `tasks.fanout`. The Host-facing surface is one role-to-model map passed to `prepare`. A chain's current model starts on `default`, persists across the chain's sections, and is copied into children when they are dispatched. Failures surface as parse errors, `prepare` refusals, or run-time errors that name the five roles. + +- Actors and workflows: + - Terms: a chain is the engine's unit of sequential execution, which is the H1 pass, the root walk, a `call` child, or a spawned task, and each fanout arm is a spawned task (`crates/runvil-internal/engine/src/execute/scheduler/chain.rs`). The H1 pass runs the Lua blocks under the prompt's title before the first section (`crates/runvil-internal/engine/src/execute/scheduler/h1.rs`). A section VM is the Lua VM built for one section. A handle is the Lua value that `models.use`, `models.get` and `models.current` return. + - Prompt author: writes optional `models:` requirements, then uses `models.current()`, `models.use(role or handle, options?)`, `models.get(role)`, `models.infer` and `tasks.fanout`. + - Host: calls `prepare` with a role-to-model map and handles the returned `Requirements`. + - A chain starts on the `default` role. `models.use` switches it for the rest of that chain, across its sections. + - `call` children, `tasks.spawn` tasks and fanout arms start with a copy of the parent's current model taken when the call or spawn is dispatched. The root walk starts with the model H1 ended on. A child's switch never reaches its parent or its siblings. +- Inputs and outputs: + - Front matter `models:` maps role names to `{ min_context, thinking }`. `min_context` is a positive token count that the model filling the role must meet. `thinking: true` requires a model that can think and turns thinking on for that role's requests. `thinking: false` requires a model that can run without thinking and turns it off. + - Host map: `BTreeMap`. An empty map means the Host has no model. A non-empty map must hold `default`. A role without an entry uses the `default` entry. + - The model handle exposes `label`, `model_id`, `description`, `context`, `thinking` and `provider`, plus the methods `infer` and `loop`. + - Lua syntax, every affected form: + + ```lua + -- FRONT MATTER (YAML shown as comments) + -- (no models: key) valid: the five roles exist + -- models: { strong: { min_context: 200000 } } valid: checked against the model filling strong + -- models: { strong: { thinking: true } } valid: model must be able to think; strong's requests think + -- models: { small: { thinking: false } } valid: model must run without thinking; small's requests do not + -- models: { fast: {} } valid, adds nothing + -- models: { reviewer: {} } PARSE ERROR: unknown role, expected default, strong, small, fast or creative + -- models: { strong: { tier: 1 } } PARSE ERROR: unknown field, expected min_context or thinking + + -- READ the chain's current model (a chain starts on the `default` role) + local m = models.current() -- handle: m.label, m.model_id, m.description, m.context, m.thinking, m.provider + -- errors "the caller supplied no model" when the Host map is empty + + -- SWITCH the chain's current model (persists across this chain's sections) + local m = models.use("strong") -- returns the handle + models.use("fast", { temperature = 0.2, max_tokens = 512 }) + models.use("default") -- the `default` role (not a fanout arm's given model) + models.use(prev) -- also accepts a handle; options apply on top of it + models.use("reviewer") -- runtime error: not a model role, lists the five + + -- INSPECT or CALL without switching + local s = models.get("small") + models.infer(prose) -- on the current model + models.infer(s, prose) -- explicit handle, frozen binding + + -- SAVE AND RESTORE around a block of work + local prev = models.current() + models.use("creative") + local drafts = tasks.fanout(worker, items) + models.use(prev) + + -- FANOUT + tasks.fanout(worker, items) -- arms inherit the chain's current model + tasks.fanout(models.get("fast"), worker, items) -- 3-arg: model handle first, parent not switched + -- inside an arm: models.use("strong") affects only that arm and its children + ``` + +- States and validation: + - Parse time: an unknown role name, an unknown field, a duplicate role key, or `min_context: 0` is a parse error. The unknown-role error lists the five names. + - `prepare` time: each declared role's requirements are checked against the model that fills it, fallbacks included. + - Run time: `models.use` and `models.get` reject a label that is not one of the five, except that `models.get` resolves a raw gateway model id when the run has an application-state snapshot (`crates/runvil-internal/lua/src/models.rs`). `models.use`, `models.get` and `models.current` on a role with no model fail with "the caller supplied no model". +- Errors and recovery: + - `prepare` reports unmet requirements, and a non-empty map without `default`, in `Requirements`. The caller refuses the run before it starts, as it does for other unmet requirements (`crates/runvil-internal/engine/src/execute/requirements.rs`). The Host recovers by supplying a model that meets the requirement, or a `default` entry. + - With an empty map, using a model raises `ModelRequired`, and a prompt that never calls a model still runs. + - A 3-arg fanout restores the parent's model before its first join and also when a spawn fails part-way. +- Security and privacy behavior: + - Chain isolation only: each chain owns its selection, so a child cannot change its parent's or a sibling's model. No new data crosses the Lua sandbox, and the handle fields listed above are all it exposes. +- Acceptance criteria: + - A prompt with no `models:` calls `models.infer` on a Host with one `default` model and succeeds. + - `models:` naming anything other than the five roles fails to parse with an error listing them. + - With the Host map `{ default: A, strong: B }`, `models.use("strong")` makes the next `models.infer` run on B, and `models.use("fast")` runs it on A. + - A switch made in one section is still in effect in the next section of the same chain. + - A parent that switches after spawning but before a queued task is admitted does not change that task's model. + - `tasks.fanout(models.get("fast"), worker, items)` runs every arm on `fast` and leaves the parent's model as it was. + - A non-empty Host map without `default` is refused before the run starts. + + + + +## Technical Design + +The Host's role map enters `prepare`, which resolves the five roles, checks requirements, and builds an immutable model set shared by the whole run. Each chain owns its current model, and children receive a copy when they are dispatched. Section VMs receive the chain's selection at Lua injection, and the engine reads it through one accessor. The 3-arg fanout is implemented in the Lua shim with no protocol change. + +- Architecture: + - `prepare` resolves the five roles from the Host map (the role's entry, else the `default` entry), checks declared requirements against the resolved models, and builds the run's `ModelSet`. The set is immutable and shared as `Arc`. Today it sits behind a mutex and a `ModelView` trait only because `models.default` writes to it; the `RunState` documentation names that as its only writer (`crates/runvil-internal/engine/src/execute/config.rs`, `fill.rs`, `context-bound.rs`, `context.rs`, `crates/runvil-internal/model-client/src/model/options.rs`). + - Each chain owns its current model as `Arc>>`. Root chains start on the `default` binding, or on `None` when the Host map is empty. `ChainStart` carries the starting value. Call and spawn dispatch copy the parent's value into the child's `ChainStart`, and `end_h1` writes the value H1 ended on into the root walk it starts (`crates/runvil-internal/engine/src/execute/scheduler/chain.rs`, `dispatch.rs`, `tasks-spawn.rs`, `h1.rs`). + - The copy happens at dispatch because a queued task holds no VM until the scheduler admits it, so a copy at admission would let a later switch in the parent leak into it (`crates/runvil-internal/engine/src/execute/scheduler/tasks-spawn.rs`). + - Each section VM receives its chain's selection as an argument to injection, which is where the `models` table is installed (`crates/runvil-internal/lua/src/vm/inject.rs`, `crates/runvil-internal/engine/src/execute/section_vm.rs`). The engine reads the current model through `SectionVm::current_model()` for inference, chat rounds and a section's first tool call (`crates/runvil-internal/lua/src/vm/state.rs`, `crates/runvil-internal/engine/src/execute/scheduler/dispatch.rs`, `chat.rs`, `crates/runvil-internal/engine/src/execute/section_context.rs`). The selection holds the whole `ModelBinding`, which lets `models.use` accept a handle, including a raw-id handle from `models.get`. + - The 3-arg fanout lives in the Lua shim (`crates/runvil-internal/lua/src/__impl_fanout.lua`). It sets the parent's selection to the given handle's binding, spawns every arm, and restores the selection before the first join and before re-raising on a spawn failure. The shim already spawns every arm before any join, no author code runs in the parent between the switch and the restore, and `models.use` emits no event, so the switch is unobservable. The shim reads and sets the selection through privileged captures installed with it (`crates/runvil-internal/lua/src/coro.rs`). + - Data flow: + + ```mermaid + flowchart TD + hostMap[Host role map] --> prep[prepare] + prep -->|"refusals"| reqs[Requirements] + prep -->|"5 bindings"| set[ModelSet] + set --> h1[H1 chain] + h1 -->|"value at end_h1"| root[Root walk] + root -->|"copy at call"| callChild[call child] + root -->|"copy at spawn"| arm[fanout arm] + ``` + +- Modules and interfaces: + - `ModelRole` enum in `crates/runvil-internal/types/src/models.rs`, re-exported by the facade in `crates/runvil/src/lib.rs`. The parser, the Lua crate and the engine already depend on `runvil-types`. + - Parser (`crates/runvil-internal/parser/src/contract/models.rs`, `contract.rs`): `RoleRequirements { min_context, thinking }`, which is the parser's existing `ModelRole` struct under a new name and frees `ModelRole` for the enum. `Frontmatter::role_requirements(ModelRole)` is the only accessor. The shared contract-map visitor becomes generic over the key type, so `args` keeps alias-grammar string keys and `models` uses `ModelRole`, with duplicate-key rejection shared. + - Engine: `RunContext::prepare`, `RequirementCheck` as `{ ContextMinimum, Thinking, Unbound }`, and `UnmetRequirement.role: ModelRole`. A declared role's `thinking` also sets the thinking switch in that role's `ModelInvocation`, as `bound_model_set` does for the keyword today (`crates/runvil-internal/engine/src/execute/context-bound.rs`). + - Lua (`crates/runvil-internal/lua/src/models.rs`, `models-userdata.rs`): `models.use`, `models.current` and `models.get` as specified above, and the handle without a `capabilities` field. + - Host API: + + ```rust + // ModelRole lives in runvil_types::models, re-exported as runvil::model::ModelRole. + enum ModelRole { Default, Strong, Small, Fast, Creative } // ALL, as_str(); Deserialize (kebab-case) + + // RunContext::prepare(self, prompt: &Prompt, models: BTreeMap) + // -> (RunContext, Requirements) + ctx.prepare(&prompt, BTreeMap::new()) // model-less Host + ctx.prepare(&prompt, BTreeMap::from([(ModelRole::Default, model)])) // one model for all five + ctx.prepare(&prompt, BTreeMap::from([(ModelRole::Default, d), (ModelRole::Strong, s)])) + // A non-empty map without Default adds + // UnmetRequirement { role: ModelRole::Default, check: RequirementCheck::Unbound, .. } + // and requirements.refusal() fails the run, like any other unmet requirement. + ``` + +- File and public API changes: + - Added: `ModelRole` with `ALL` and `as_str()`; `RoleRequirements` with `min_context()` and `thinking()`; `Frontmatter::role_requirements`; `RequirementCheck::Thinking` and `RequirementCheck::Unbound`; Lua `models.current`, `models.use(handle)` and the 3-arg `tasks.fanout`. + - Changed: the signature of `RunContext::prepare`; the type of `UnmetRequirement.role` from `String` to `ModelRole`; the Workshop prompt-contract JSON model entries become `{ label, min_context, thinking }` (`crates/workshop/server/src/routes/prompts.rs`), read by `crates/workshop/ui/src/services/run-api.ts` and `crates/workshop/ui/src/parts/run/run-rows.ts`. + - Deleted from the facade's public API (`crates/runvil/public-api.txt`): `ModelBindings` with its five methods and `RunContext::model_bindings()`; `ModelRoles`, `ModelRolesIter` and `Frontmatter::models()`; `ModelKeyword`; `ModelBinding::capabilities` and `with_capabilities`; `RequirementCheck::HardKeyword`. + - Deleted from the Lua and front matter surface: `models.default`, the handle's `capabilities` field, front matter `keywords:` and per-role `description:`. + - Deleted across modules: `ModelView`, `ModelSetError`, the default alias and its accessors on `ModelSet`, the run-wide model-set mutex, `ModelSelection` and `resolve_model_binding`. + - Regenerate `crates/runvil/public-api.txt`. + - Rewrite in place: `README.md`; `prompts/hello.md`, `prompts/greet.md`, `prompts/research-person.md`, `prompts/analyst-example.md`; `crates/workshop/agents/agents/chat.md`; about 30 fixtures that embed `models:` front matter under `crates/runvil-internal/engine`, `crates/harness-internal/runner/tests/it`, `crates/harness/tests/suite`, `crates/runvil/tests/suite`, `crates/harness-gateway-client/src/wire/request-tests.rs` and `crates/workshop/server`; and the models section of `runvil-design/design/architect-runvil-prompting-language.md` in the sibling design repository. +- Data, persistence, failure, security, and privacy constraints: + - The only externally visible data change is the Workshop prompt-contract JSON above. No other persisted format changes. + - `prepare` refuses a run before any effect is issued: unmet requirements and a missing `default` are carried in `Requirements`, and the harness only merges and refuses (`crates/harness-internal/runner/src/prepare.rs`). No non-test code outside `requirements.rs` and the constructing code in `fill.rs` matches on `RequirementCheck` or reads `UnmetRequirement.role` or `.check`, so those types can change freely. + - The harness converts its `Option` from `current_model` into the Host map at one place, `prepare_from_snapshot`: an empty map for `None`, `{ default: m }` for `Some(m)` (`crates/harness-internal/runner/src/environment.rs`, `crates/harness-internal/runner/src/prepare.rs`). + - Chain isolation: no two chains share a selection, and inheritance copies a value. + + + + +## Testing Plan + +Tests cover the parse rules, `prepare` resolution, the Lua surface, chain inheritance and the 3-arg fanout, each in the step that introduces the behavior. Two sweeps check that deleted names and backward-looking wording are gone. Each component ends with a gate of full tests, lints and the sweeps. + +- Unit: + - Types: `ModelRole` deserializes each of the five kebab-case names, and an unknown name's error lists all five. + - Parser: no `models:`; each role declared; `min_context` and `thinking` read back; an unknown role rejected with the five names; an unknown field (`tier`) rejected; a duplicate role key rejected; `min_context: 0` rejected; `args` keys still checked against the alias grammar. + - `prepare`: + - a one-model map gives all five roles that model; + - a map with a `strong` entry gives only `strong` that model; + - `min_context` on a fallback role is checked against the default model; + - `thinking: true` against a model whose thinking mode is `Never`, and `thinking: false` against one whose mode is `Always`, are each refused; + - a non-empty map without `default` is refused with `Unbound`; + - an empty map leaves the set empty, and `models.infer` raises `ModelRequired`; + - a declared `thinking` sets the thinking switch on that role's requests. + - Lua: + - a prompt with no `models:` can `models.use("strong")`; + - `models.current` reflects `models.use`; + - `models.use(handle)` works with options applied on top; + - an unknown label errors with the five names listed; + - a role with no model errors with the Host-supplied-none message; + - the handle exposes `label`, `model_id`, `description`, `context`, `thinking` and `provider`. +- Integration and end-to-end: + - Chain selection (second component): + - a switch persists across a chain's sections; + - a `call` child and a spawned task each inherit the selection; + - a child's switch does not reach the parent or a sibling; + - a switch in H1 reaches the root walk; + - a parent that switches after spawning but before a queued task is admitted does not change that task's model; + - all arms of one fanout start on the same model. + - Fanout (third component): + - a 3-arg fanout runs every arm on the given model; + - the parent's model after the call is what it was before; + - a spawn failure part-way still restores it; + - a raw-id handle works with an empty Host map; + - a non-handle first argument is an error. + - A prompt with no `models:` runs end to end through the harness against a one-model Host, using the rewritten fixtures. +- Regression, security, and performance: + - Deleted-name sweep: search `crates`, `README.md` and `prompts` in the runvil repository, and `design` in the sibling design repository, for `models.default`, `models.bind`, `default_alias`, `set_default_alias`, `ModelView`, `ModelSetError`, `ModelBindings`, `model_bindings`, `ModelRoles`, `ModelRolesIter`, `ModelKeyword`, `keyword_name`, `keyword_wire`, `with_capabilities`, `HardKeyword`, `resolve_model_binding`, `trivial fill` and `bound model role`. Expect no hits. Skip `vibe/` and `.vibelight/`. + - Wording sweep: search the added lines of the diff for `no longer`, `formerly`, `previously`, `legacy`, `deprecat`, `backward`, `replaces` and `migrat`. Expect no hits. + - Public API check: `crates/runvil/public-api.txt` is regenerated and matches the surface listed in Technical Design. + - Security: the child-isolation tests above. + - Usage attribution: with the Host map `{ default: A, strong: B }`, the events of a `strong` round carry B's model name. Events take their `model` field from the completion (`crates/runvil-internal/engine/src/execute/turn.rs`, `crates/runvil-internal/types/src/event.rs`). + - Performance: no requirement. +- Exit criteria: + - `cargo test` and clippy pass for the types, parser, model-client, lua, engine, harness, facade and Workshop server crates, and the Workshop UI builds. + - Both sweeps return no hits and the public API check passes. + - Each component meets these criteria at its last step before the next component begins. + + + + +## Decision Record + +- Decisions: + - Exactly five fixed roles and no custom roles. Roles name tiers the Host maps to models, and a custom label added only ceremony because the Host bound every label to the same model. User: "default, strong, small, fast, creative" and "no custom roles". + - `thinking` is a requirement on a model, not a role. User: "\"thinking\" is a property of a model not a model role". + - No `compact` role yet. User: "for compact we defer because we dont have enough evidence to know what is needed." + - `models:` is optional and carries only `min_context` and `thinking`. With fixed role names the soft keywords and per-role `description` had no runtime effect; they fed only the handle's `capabilities` and `description` fields and the Workshop run panel text (`crates/runvil-internal/engine/src/execute/context-bound.rs`, `crates/workshop/ui/src/parts/run/run-rows.ts`), and the two hard keywords are one tri-state. User: "no prompt that lacks a \"models:\" section should fail"; from the maintainers' design discussion, "we dont want to remove the possibility of role requirements, just avoid the need for the typical prompt to have to write" a `models:` block; and the user chose the option of only `min_context` and `thinking`. + - The Host passes a plain `BTreeMap`, required in every case, empty allowed, and a non-empty map must hold `default`. A role without an entry uses `default`. A missing `default` is refused through `Requirements` as `Unbound`, and the keys are the closed `ModelRole` enum so a typo cannot fall back silently. User: "Just require a map in all cases. And the map can be empty. if it is not empty, it must have at least \"default\"" and "the harness will use the default model for any missing role"; the user chose the plain map and the closed role type. + - The current model is chain state. A chain starts on `default` and a switch persists until the next switch. User: "a chain should start with \"default\" and that stays until the model is switched. models.current() returns the current model object, models.use() switches roles and returns the model object." + - `models.default` is deleted and the `default` role is the baseline. The call was the only writer of the shared model set, which is why the set needed a mutex. The idea came from the maintainers' design discussion, where David Rauschenbach wrote "maybe we should get rid of that, and add a \"default\" role", and Jason agreed with dropping `models.default`. + - Children inherit a copy of the parent's current model, taken at dispatch. User: "fanout inherits the chain's current model". Taking the copy at dispatch keeps a queued task's model independent of when the scheduler admits it. + - A 3-arg `tasks.fanout(model, worker, collection)` with the model handle first, implemented in the Lua shim. User: "maybe we add a 3-arg fanout which puts the model object first". The shim switch-and-restore needs no protocol or scheduler change and spares authors a manual save and restore. + - `models.use` accepts a handle as well as a role label. It comes from the design discussion's `model.use(prev)` snippet and keeps the save-and-restore pattern working. It was proposed during planning and kept in every revision of this plan without objection. + - No backward compatibility, and no trail to old behavior in code, docs, errors or tests. User: "I want no backward compatibility. Dont leave behind notes referring to the old behavior, etc.. just change it." + - Reduce the API surface while keeping the results. User: "see if you can reduce the API surface and get the same results". Applied as: an immutable `ModelSet`, `prepare` building the model set directly so `ModelBindings` goes, and the selection storing a whole `ModelBinding`. + - `Frontmatter::role_requirements(role)` returns `Option<&RoleRequirements>`: `None` when the prompt does not declare the role, and `Some` with neither field set when it declares `role: {}`. The Workshop prompt contract lists exactly the declared roles, which needs that distinction. The plan named the accessor without its return type. + - The work is built as three components in eight steps. Each step is one commit that leaves the workspace building with its tests passing, and the component that ends at a step carries that step's gate of full tests, lints, the public API check and both sweeps. The eight steps are: (1) the `ModelRole` type; (2) the Host role map in `prepare`; (3) the immutable model set with the `default` role as baseline; (4) `models.use` with a handle, `models.current` and the handle fields; (5) requirements-only front matter with the Workshop contract and UI, which ends the first component and its gate; (6) the chain-owned current model with inheritance at dispatch, which ends the second component and its gate; (7) the model-first `tasks.fanout`, which ends the third component and its gate; (8) one design-document step in `runvil-design`, written after all the code, that describes the final surface of all three components. The rationale is fewer review cycles, and a document that describes the finished surface once instead of three partial states. The owner said "12 steps seems like a lot" about a split that gave each component its own design-document commit, and chose this eight-step shape. The three components are independently shippable, and only the second carries real risk. +- Rejected alternatives: + - Custom prompt-declared roles alongside the five. The user rejected them. Revisit only if the five tiers prove insufficient, which would be a change to the prompt language. + - A `thinking` role. The user rejected it because thinking is a model property. + - Keeping `prepare` on a single optional model. Rejected because per-role models are the purpose of the map. No revisit condition. + - A required default-model parameter in place of a map. Rejected because the harness supports Hosts with no model, where `current_model` returns `None` for no selection and an empty catalog (`crates/harness-internal/runner/src/environment.rs`), and such a Host must still run prompts that call no model. Revisit if the harness stops supporting model-less runs. + - A `RoleModels` newtype whose constructors make a map without `default` unrepresentable. Rejected for the plain map and an `Unbound` refusal, which adds no type, error or constructors. Revisit if Hosts build maps in many places and want the rule enforced by construction. + - `FromStr` and an `UnknownRole` error for `ModelRole`. Rejected because the serde derive already lists the five names. Revisit when a Host reads role names from configuration. + - Rolling the selection forward like `var` at each section exit. Rejected for a chain-owned selection, because rolling forward needs a read-back at about five exit sites (`end_section`, `apply_jump`, chain `finish`, `step.rs`, `end_h1` under `crates/runvil-internal/engine/src/execute/scheduler`). No revisit condition. + - A model field on the spawn request, and a separate setter on the VM called before injection. Rejected for the shim switch-and-restore and an injection argument, which is the only place the `models` table is installed. Revisit if a consumer needs the fanout's model switch as an event. + - Keeping the soft keywords, `description` and the handle's `capabilities`. Rejected as dead surface. Revisit if a Host needs role metadata. + - A `models.default` stub or a migration error. Rejected by the no-compatibility decision. +- Assumptions, risks, and notes: + - Paths in this plan are relative to the root of the `runvil` repository, except paths starting with `runvil-design/`, which are in the sibling design repository. + - "No prompt that lacks a `models:` section should fail" is read as a statement about front matter. With an empty Host map a prompt fails only when it calls a model, which follows from "the map can be empty". + - Model-call events carry a `model` field set from the completion, not from the request (`crates/runvil-internal/engine/src/execute/turn.rs`, `crates/runvil-internal/types/src/event.rs`), so attribution by model is available. Whether `Completion::model()` falls back to the requested model when a backend omits the name was not established, and events then carry a `ModelMetadataDegraded` signal. A test pins the attribution rather than assuming it. + - The search for `RequirementCheck` consumers skipped gitignored paths and the Workshop UI `dist` bundles, and a read of `role` or `check` through JSON would not have matched the type names. It found no non-test consumer. + - Persistence is a cost footgun: a section that calls `models.use("strong")` for one expensive step leaves every later section of that chain on `strong`. The behavior is documented and tested, and a scoped form can follow. + - Requirement failures reflect fallbacks: `strong: { min_context: 200000 }` is refused on a Host whose default model is smaller and that supplies no `strong` entry. That is intended and can surprise a Host author. + - The `model` field of the Lua `sys` table is fixed at a section's first tool call (`crates/runvil-internal/engine/src/execute/section_context.rs`), so with chain-level persistence it shows the chain's current model at that moment. + - `creative` may overlap with the `temperature` option on `models.use` (`crates/runvil-internal/lua/src/models.rs`). It stays because the user chose it as the fifth role. + - Commit messages describe the change as usual. The no-trail constraint covers code, docs, error messages and tests. The user did not answer whether it also covers commit messages. + - Confidence: medium-high. Every seam was checked against the code, and what remains is the size of the first component and the event-attribution check. + +### Deferred and Out of Scope + +- Deferred: a role for compaction. Revisit when evidence shows what `models.compact` needs (`crates/runvil-internal/lua/src/models.rs`). +- Deferred: per-role model selection in production Hosts. The harness passes only `default` because its host snapshot carries one `selected_model` (`crates/harness-internal/runner/src/environment.rs`). Revisit when that snapshot carries per-role selections. +- Deferred: a scoped model-switch form. Revisit if chain-level persistence leaks cost in practice. +- Deferred: `FromStr` for `ModelRole`. Revisit when a Host reads role names from configuration. +- Out of scope: the handle's duplicate `name` and `label` fields (`crates/runvil-internal/lua/src/models-userdata.rs`). +- Out of scope: a model argument on `tasks.spawn`. +- Out of scope: editing dated logs under `vibe/` and the generated `.vibelight/` directory. + + + + +## Project Survey + +- Status: complete +- Build command: `cargo build --locked` builds the default members (`runvil`, `runvil-plugin`, `harness`, `harness-gateway-client`) with no native toolkit. The desktop app is `cargo build --locked -p workshop`, which needs `npm ci --prefix crates/workshop` and a staged Gateway sidecar (`node tools/fetch-gateway-sidecar.mjs --stub --target `, removed afterward with `node tools/stage-gateway-sidecar.mjs remove --target `). The UI bundle alone is `npm run build --workspace ui`, run in `crates/workshop`. +- Focused test command pattern: `cargo nextest run --locked -p `. A single UI test file is `node --test crates/workshop/ui/test/.mjs`. The structural checks are `cargo test -p build-xtask`. +- Component test command pattern: `cargo nextest run --locked -p ` for a Rust crate, and `npm test --workspace ` run in `crates/workshop` for a UI package. +- Full-suite test command: `cargo nextest run --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-features`, then the workshop crates separately with `cargo nextest run --locked -p workshop -p workshop-server -p workshop-server-api`. UI: `npm test --workspaces --if-present` in `crates/workshop`. Wording scan: `node --test crates/workshop/ui/test/docs-claims.mjs`. +- Linter command: `cargo clippy --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-targets --all-features`, and for workshop `cargo clippy -p workshop -p workshop-server -p workshop-server-api --all-targets`. Both run with `CARGO_BUILD_WARNINGS=deny` in the gate. The UI also has `npm run typecheck --workspaces --if-present` in `crates/workshop`. +- Formatter check command: `cargo fmt --all --check` +- Docs command: `cargo doc --workspace --no-deps --all-features --exclude workshop --exclude workshop-server --exclude workshop-server-api` with `RUSTDOCFLAGS="-D warnings"`. The facade docs are `cargo doc -p runvil --no-deps` and `cargo doc -p harness --no-deps`, both without `--all-features`. The public surface check is `cargo + xtask api --check`, where the nightly is named in `crates/build-xtask/src/api/toolchain.rs`. +- Test placement and naming conventions: a unit test lives in a sibling file named `-tests.rs`, wired as `#[cfg(test)] #[path = "-tests.rs"] mod tests;`, or in a `tests/` module directory beside the source when it has three or more files. An integration suite is one directory per binary, `crates//tests//main.rs` (for example `crates/runvil/tests/suite`, `crates/harness/tests/suite`, `crates/plugin-std/tests/it`) with one topic file per area, and prompt fixtures sit in `tests/prompts`. Test functions are `#[test]` with long sentence-style snake_case names that state the behavior. UI tests are `.mjs` files run by `node --test`, in `crates/workshop/ui/test/` and as `src/**/*.test.mjs`. Benches are `harness = false` and need the crate's `test-support` feature. Tests under `tools/` are `.test.mjs`. +- Directory map: + - `crates/` holds the shared and public layer: `runvil` (Engine facade), `runvil-plugin`, `harness`, `harness-gateway-client`, `plugin-mcp`, `plugin-std`, `plugin-user-input`, `plugin-web`, `shared-error-source`, and the build crates (`build-ceiling`, `build-ui`, `build-user-guide`, `build-workshop`, `build-xtask`). + - `crates/runvil-internal/` holds the Engine's private crates: `engine`, `lua`, `model-client`, `parser`, `types`, `vfs`. + - `crates/harness-internal/` holds the Harness's private crates: `runner`, `plugin-loader`, `test-plugin`. + - `crates/workshop/` holds the Workshop crates (`agents`, `desktop`, `gateway`, `gateway-discovery`, `menu`, `protocol`, `registry`, `run-log`, `server`, `server-api`, `status`, `support`, `user-state`, `workspace`) and the npm workspace members `ui`, `look`, `platform`. + - `guide/` holds the books (`language`, `workshop`), the site chrome, and the landing page. + - `prompts/` holds the sample prompts. `tools/` holds the Node scripts for the Gateway pin and sidecar. `gateway-pin/` holds the pinned Gateway release (`pin.json`). + - `vibe/` holds dated plan notes, with `scratch/` ignored. `local/` holds local-only files. `.vibelight/` holds generated crate outlines. `.github/` holds the CI, nightly, release, installer-smoke, and site workflows. `.githooks/` holds `pre-commit` (format check) and `pre-push` (wording scan, clippy, `cargo deny`). `.cursor/rules/` holds the Workshop rules. `.config/nextest.toml` holds the nextest profiles. `images/` holds README images. +- Component boundaries: the Engine is `runvil-types` and `runvil-vfs` at the bottom, then `runvil-model-client`, `runvil-lua`, `runvil-parser`, and `runvil-engine`, fronted by `runvil` and `runvil-plugin`. Only those two crates may depend on `crates/runvil-internal`. The Harness is `harness-runner` and `harness-plugin-loader` behind `harness`, which depends on `runvil` and `runvil-plugin`, and `harness-gateway-client` sits on `harness` and `plugin-web`. Each Pack crate (`plugin-*`) depends only on `runvil-plugin`. The Workshop tiers run one way, server to features to services to vocabulary: `workshop-server` composes the feature and service crates. `workshop-agents` and `workshop-run-log` sit on `harness`, and the rest sit on `workshop-registry`, `workshop-support`, and `workshop-protocol`, with `workshop-registry` also depending on `workshop-protocol`. The desktop app `workshop` depends only on `workshop-server-api`, which fronts `workshop-server`. Engine crates never depend on `tokio`, `reqwest`, or the other crates the structural rules ban. Code in one family reaches another only through a crate at the `crates/` level, and `cargo test -p build-xtask` enforces the boundaries. +- Conventions summary: the Rust edition is 2024 on a pinned stable 1.99 toolchain with an MSRV of 1.95, and `rustfmt` uses `style_edition = "2024"`. Workspace lints deny `unsafe_code`, `unwrap_used`, `expect_used`, `todo`, and clippy `all` and `pedantic`, and `allow_attributes` is denied, so a suppression uses `#[expect(..., reason = "...")]`. Source directories are flat, and a subdirectory needs at least three files, otherwise the files sit beside the parent as `foo-bar.rs` with an explicit `#[path]`. Every `workshop-*` `lib.rs` is a facade whose `//!` doc carries a `## Invariants` marker. The seven defined terms (Engine, Harness, Host, Pack, Plugin, Static Set, Dynamic Set) are capitalized and each has one meaning, enforced by `docs-claims.mjs`. Engine crates call their caller "the caller" and never name the Host. Behavior changes ship with tests in the same change. Error messages are written for a model to read. Files use LF endings, and Windows links with `rust-lld` and a static CRT. + + + + +## Execution Instructions + + + +### Step 1: ModelRole type [completed] + +- Component: Fixed roles and Host map +- Tests: + - `cargo nextest run --locked -p runvil-types model_role` + - `cargo fmt --all --check` + - `$env:CARGO_BUILD_WARNINGS = 'deny'; cargo clippy --locked -p runvil-types --all-targets` + - `git add -N .; git diff -U0 HEAD | rg -n -i "^\+.*(no longer|formerly|previously|legacy|deprecat|backward|replaces|migrat)"` (a pass prints nothing and exits with code 1) + +Add the closed set of five roles to `runvil-types`. + +- Placement: this component goes first because the other two build on its immutable model set, its selection argument and the Host role map, and the third needs the second's copy at dispatch. This piece, the role type, is built first and on its own because the parser, model-client, Lua, engine and facade all name it and it needs none of them. +- Build state: the workspace builds and passes after this step. Nothing outside `runvil-types` changes. +- Artifacts: in `crates/runvil-internal/types/src/models.rs` add the enum `ModelRole` with variants `Default`, `Strong`, `Small`, `Fast`, `Creative`. Add `ModelRole::ALL: [ModelRole; 5]` in that order and `ModelRole::as_str(self) -> &'static str` returning `default`, `strong`, `small`, `fast` and `creative`. Derive `Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, Deserialize` with `#[serde(rename_all = "kebab-case")]`, and add `#[non_exhaustive]` the way `ThinkingMode` has it, so the type works as a `BTreeMap` key. Do not add `FromStr`, `Display` or an unknown-role error type, because the serde error already lists the five names. The doc comment says each role is a tier the Host maps to a model, and that a role with no model of its own uses `default`. Put the tests in the file's existing `mod tests`. +- Test cases (sentence-style test names that contain `model_role`): + - each of `"default"`, `"strong"`, `"small"`, `"fast"` and `"creative"` deserializes with `serde_json::from_str` to its variant; + - `ModelRole::ALL` lists the five in the order above, and each role's `as_str()` text deserializes back to the same role; + - the name `"reviewer"` fails to deserialize, and the error text names all five of `default`, `strong`, `small`, `fast` and `creative`. +- Rules: nothing you write (code, comments, docs, error messages, test names) says "replaces", "formerly", "previously", "no longer", "legacy", "removed" or "migrated", or notes what a name used to be, and no test asserts that a deleted name is gone. Do not edit `vibe/` or `.vibelight/`. + + + + + +### Step 2: Host role map in prepare [completed] + +- Component: Fixed roles and Host map +- Tests: + - `cargo check --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-targets --all-features` + - `cargo check --locked -p workshop-server --all-targets` + - `cargo nextest run --locked -p runvil-engine -p runvil -p harness-runner -p harness -p harness-gateway-client --all-features` + - `cargo nextest run --locked -p workshop-agents` + - `cargo nextest run --locked -p build-xtask` + - `cargo fmt --all --check` + - `$env:CARGO_BUILD_WARNINGS = 'deny'; cargo clippy --locked -p runvil-engine -p runvil -p harness-runner -p harness --all-targets --all-features` + - `cargo +nightly-2026-09-05 xtask api --check` + - `node --test crates/workshop/ui/test/docs-claims.mjs` + - `rg -n "ModelBindings|model_bindings|bind_role_for_test|bind_roles|HardKeyword|trivial fill" crates README.md prompts` (a pass prints nothing and exits with code 1) + - `rg -n "models\.default" prompts README.md crates/runvil crates/harness crates/harness-internal crates/harness-gateway-client crates/workshop` (a pass prints nothing and exits with code 1) + - `git add -N .; git diff -U0 HEAD | rg -n -i "^\+.*(no longer|formerly|previously|legacy|deprecat|backward|replaces|migrat)"` (a pass prints nothing and exits with code 1) + +Change `RunContext::prepare` to take the Host's role map, so five roles exist for every prompt and a Host can give each its own model. The parser keeps reading `models:` with its current label grammar until step 5, so in this step prompts and fixtures use the five role names as their labels. + +- Placement: this piece, "Host map to model set", is steps 2 to 4, built one after another. The plan treats the engine, model-client and Lua changes as one compile unit, but that holds only if the front matter reader changes at the same time. Keeping the parser's reader until step 5 lets each of the three steps build and pass on its own. The piece comes before the front matter piece because the front matter's requirements are checked against the models this piece resolves. +- Build state: the workspace builds and passes after this step. The model set is still behind its mutex and still has its default alias, which step 3 removes. +- Behavior: + - `prepare(self, prompt: &Prompt, models: BTreeMap) -> (RunContext, Requirements)`. A role's model is its own map entry, else the `default` entry. An empty map means the caller has no model: the model set stays empty and nothing is checked. A non-empty map without a `default` entry adds one unmet requirement `UnmetRequirement { role: ModelRole::Default, check: RequirementCheck::Unbound, required: "a model", actual: "none" }`, and the model set stays empty. + - Declared requirements are read with the parser's current accessor, `prompt.frontmatter().models().get(role.as_str())`, for each role in `ModelRole::ALL`. A declared label that is not one of the five is ignored in this step. `min_context` above the context of the model that fills the role is `RequirementCheck::ContextMinimum` with `required` the minimum and `actual` the model's context, both as decimal text. The hard keyword `thinking` against `ThinkingMode::Never`, and `no-thinking` against `ThinkingMode::Always`, is `RequirementCheck::Thinking` with `required` set to `thinking: true` or `thinking: false` and `actual` set to `Never` or `Always`. Soft keywords and `description` are not checked. + - The model set holds five `ModelBinding`s in `ModelRole::ALL` order, labelled with `as_str()`, for every prompt whether or not it declares the role. Build each as `bound_model_set` builds one today (the declared keywords become `capabilities`, a hard keyword sets `ModelInvocation::new().with_thinking(Some(true))` or `Some(false)`, the declared description else the model info's, `info.context()`, and `.with_catalog_info(info.clone())`). When the set has a `default` binding, call `set.set_default_alias("default".to_owned())` so a section that never calls `models.use` runs on it. Step 3 replaces that call. + - The refusal notice lines, exactly: `- role 'strong': requires a context of at least 200000 tokens; the model filling it provides 32000`, then `- role 'strong': requires 'thinking: true'; the thinking capability of the model filling it is Never`, then `- role 'default': requires a model; the caller's role map has no 'default' entry`. Engine text says "the caller" and never names the Host. +- Artifacts (Engine): + - `crates/runvil-internal/engine/src/execute/config.rs`: the new `prepare` signature. Replace the field `model_bindings: ModelBindings` with `model_set: ModelSet` (empty in `new`, and in the `Debug` output). Delete `RunContext::model_bindings()` and `bind_role_for_test`. Rewrite the `prepare` and `with_ui` docs for the role map and the five roles (`models.get` resolves a raw gateway model id for a label outside the five roles when a UI snapshot is set). + - `crates/runvil-internal/engine/src/execute/fill.rs`: `fill_model_set(prompt, models, &mut requirements) -> ModelSet` in place of `fill_model_bindings`, holding the build described above. Delete `bound_model_set` from `context-bound.rs`, and make the code in `RunState::new` (`crates/runvil-internal/engine/src/execute/context.rs`) that called it take the prepared set from the context. It still wraps it in `Arc>`. + - Delete `crates/runvil-internal/engine/src/execute/bindings.rs` with `git rm`, its `mod bindings;` line in `execute.rs`, and `ModelBindings` from the exports in `crates/runvil-internal/engine/src/lib.rs`. Re-export `ModelRole` from the engine's `model` module next to `ModelInfo`. + - `crates/runvil-internal/engine/src/execute/requirements.rs` and `requirements-tests.rs`: `RequirementCheck { ContextMinimum, Thinking, Unbound }`, `UnmetRequirement.role: ModelRole`, the notice lines above using `role.as_str()`, and the docs of `Requirements::unmet_models`, `UnmetRequirement` and `RequirementCheck`. + - `crates/runvil-internal/engine/src/test_support/scenario.rs`: the scenario holds `BTreeMap` for `prepare`. `Scenario::current_model(model)` inserts the `default` entry, add `Scenario::role_model(role, model)`, and delete `Scenario::bind_roles` with its field and the loop that called `bind_role_for_test`. +- Artifacts (facade and Harness): + - `crates/runvil/src/lib.rs`: in the `model` module drop `ModelBindings` and add `ModelRole`. Regenerate `crates/runvil/public-api.txt` with `cargo +nightly-2026-09-05 xtask api --bless`. `runvil::prompt::ModelRole` (the parser's struct) stays until step 5, and the two names live in different modules. + - `crates/harness-internal/runner/src/prepare.rs`: `prepare_from_snapshot` converts its `Option` into the map in this one place, an empty map for `None` and `{ default: model }` for `Some(model)`. Update the doc of the `model` field there to say it is the Host's default model. + - Every other `prepare` caller passes a map: `None` becomes `BTreeMap::new()` and `Some(m)` becomes `BTreeMap::from([(ModelRole::Default, m)])`. Find them with `rg -n "\.prepare\(" crates`. They include `crates/runvil-internal/engine` (`benches/models_loop.rs`, `src/execute/run/tests.rs`, `tests-drops.rs`, `src/execute/tests/live_infer.rs`, `src/execute/tests/suite/prepare.rs`, `src/execute/requirements-tests.rs`), `crates/runvil` (`examples/greeter.rs`, `tests/suite/prepare.rs`, `prompt.rs`, `replay.rs`, `support/scenario.rs`), `crates/harness-internal/runner/tests/it` (`support.rs`, `effect_loop-cap.rs`, `effect_loop-broker.rs`, `prepare*.rs`, `tool_context.rs`) and `crates/harness-gateway-client/src/wire/request-tests.rs`. +- Artifacts (prompts, fixtures, README): find every prompt and fixture that declares a custom label or calls `models.default` with `rg -n "models\.default|models:" crates prompts README.md`. Rewrite each in place so it works with the five roles: + - a prompt that needs no requirement drops its `models:` block and its `models.default(...)` call, and runs on the `default` role; + - a prompt that declares a requirement keeps a `models:` block keyed by a role name (`strong`, `small`, `fast`, `creative` or `default`), keeps only `min_context` and the hard keywords `thinking` and `no-thinking`, and drops soft keywords and `description:` unless a test in this step asserts on them; + - `models.use(' + + + +### Step 3: Immutable model set with the default role as baseline [completed] + +- Component: Fixed roles and Host map +- Tests: + - `cargo nextest run --locked -p runvil-model-client -p runvil-lua -p runvil-engine -p runvil -p harness-runner -p harness --all-features` + - `cargo fmt --all --check` + - `$env:CARGO_BUILD_WARNINGS = 'deny'; cargo clippy --locked -p runvil-model-client -p runvil-lua -p runvil-engine --all-targets --all-features` + - `cargo +nightly-2026-09-05 xtask api --check` + - `rg -n "models\.default|default_alias|ModelView|ModelSetError|resolve_model_binding|ModelSelection" crates README.md prompts` (a pass prints nothing and exits with code 1) + - `git add -N .; git diff -U0 HEAD | rg -n -i "^\+.*(no longer|formerly|previously|legacy|deprecat|backward|replaces|migrat)"` (a pass prints nothing and exits with code 1) + +Make the run's model set immutable, and make the `default` role the baseline a section starts on. The Lua selection now holds a whole `ModelBinding`, and the Lua `models.default` call and everything that existed only to support it are deleted. + +- Placement: second step of the "Host map to model set" piece, built after step 2 because it reads the five role bindings that step puts in the set, and before step 4 because `models.current` and `models.use(handle)` read and write the selection this step creates. +- Build state: the workspace builds and passes after this step. The facade's public API listing does not change. Each section VM still gets a fresh selection, so a switch lasts for the rest of its section only. Step 6 makes the chain own it. +- Behavior: + - `ModelSet` has no default alias and no writer, so it is built once at prepare and shared as `Arc`. `ModelView`, `ModelSetError`, the poisoned-lock helper and the run-wide mutex are deleted. + - Each section VM's injection takes the selection as an argument, of type `Arc>>`. The engine builds it per section as `Arc::new(Mutex::new(ctx.default_binding()))`, so a section starts on the `default` binding, or on `None` when the caller's map was empty. + - `models.use(label, options?)` accepts a role name. It looks the role up in the set and stores `options.apply(binding)` as the selection, then returns the handle. The option checks and their error texts stay as they are (for example `models.use option temperature must be a number, got string`). + - `models.get(label)` returns the handle for a role without selecting it. For a label that is not one of the five roles it resolves a raw gateway model id when the run has an application-state snapshot (`RunContext::with_ui`), as it does today with the 8192-token fallback context, and otherwise fails. + - Error texts: a label outside the five fails with `models.use label "reviewer" is not a model role: expected default, strong, small, fast or creative`, and `models.get label "reviewer" ...` with the same tail. A role with no model (the caller's map was empty) fails with an error that contains `the caller supplied no model`, for `models.use` and `models.get`. + - The engine reads the current model for inference, chat rounds and a section's first tool call through the new `SectionVm::current_model(&self) -> Result>`. It returns an `Error::internal` with `model selection lock was poisoned` on a poisoned lock. +- Artifacts (model-client): + - `crates/runvil-internal/model-client/src/model/options.rs`: `ModelSet` keeps `bindings()`, `binding(alias)` and `push`. Delete its `default` field, `default_alias()` and `set_default_alias()`, plus `ModelSetError`, `ModelView`, `lock_model_set` and the `ModelView` impl for `Mutex`. Rewrite the `ModelSet` doc to say it is built once at prepare and shared. + - `crates/runvil-internal/model-client/src/model.rs` and `lib.rs`: drop `ModelView` and `ModelSetError` from the re-exports and docs. + - Tests: `options-tests.rs` loses the poisoned-set tests, and `model/tests.rs` rewrites the test named for growing by push and taking a default alias so it covers push and lookup only. +- Artifacts (Lua, `crates/runvil-internal/lua/src`): + - `models.rs`: delete `ModelSelection`, `lock_models` and the `models.default` function. `install_models(lua, globals, set: &Arc, selection: &Arc>>, raw_ids)`. Validate a label against `ModelRole::ALL` (the crate already depends on `runvil-types`) instead of the alias grammar. Fix the module doc. + - `vm.rs` and `vm/inject.rs`: `bound_models: Arc`. The VM no longer creates a selection of its own. Injection takes it, and the `model_selection` field and accessor go. + - `vm/state.rs`: delete `resolve_model_binding` and add `SectionVm::current_model`. Update the exports in `lib.rs`, the `From` impl in `error.rs`, and the helpers in `test_support.rs` (`model_bag_handles`) and `test_vm.rs` (`run_injection`). + - `models-tests.rs`: rewrite for the new signatures. +- Artifacts (engine, `crates/runvil-internal/engine/src`): + - `execute/context.rs`: `RunState` holds `Arc`, `models()` returns `&ModelSet`, and add `RunState::default_binding(&self) -> Option`, a clone of the `default` binding. Delete the `ModelView` use. + - `execute/section_context-construct.rs` (it passes the model set to the VM) and `execute/section_vm.rs`: create the per-section selection as above and pass it to injection. + - `execute/scheduler/chat.rs`, `execute/scheduler/dispatch.rs` and `execute/section_context.rs`: call `vm.current_model()` where they called `resolve_model_binding(ctx.models(), vm.model_selection())`. Update the re-exports in `lua.rs` and `model.rs`. + - `execute/fill.rs`: remove the `set_default_alias` call that step 2 added. + - Tests that build a `ModelSet` by hand with a default alias push a binding labelled `default` instead: `execute/tests/scheduler.rs`, `execute/tests/models_loop.rs`, `execute/tests/precheck_anchor.rs`, `execute/tests/models_loop-author-shapes.rs`, `execute/tests/scheduler/failures-script-tools.rs`, `model/tests.rs`, `model/tests-always.rs`, `error/tests.rs`. Find any other with `rg -n "default_alias|ModelView|ModelSelection|resolve_model_binding" crates`. +- Test cases (Lua unit tests in `models-tests.rs`, engine tests through `Scenario` with the map `{ default: A, strong: B }`): + - a section that never calls `models.use` runs `models.infer` on A, the `default` role's model; + - `models.use('strong')` makes the next `models.infer` run on B, and `models.use('fast')` runs it on A; + - `models.use('strong', { temperature = 0.2, max_tokens = 512 })` applies both options to the next round, and the existing option-validation tests keep their texts; + - `models.use('reviewer')` fails with `models.use label "reviewer" is not a model role: expected default, strong, small, fast or creative`, and `models.get('reviewer')` with no UI snapshot fails with the `models.get` form of the same text; + - with an empty caller map, `models.use('default')` and `models.get('default')` fail with an error containing `the caller supplied no model`, `models.infer` raises `ModelRequired`, and a prompt that never calls a model still runs; + - with a UI snapshot, `models.get('vendor/raw-model')` resolves a raw gateway model id with the 8192-token fallback context; + - `SectionVm::current_model()` returns the `default` binding on a new section VM and the selected binding after `models.use('strong')`. +- Rules: nothing you write (code, comments, docs, error messages, test names) says "replaces", "formerly", "previously", "no longer", "legacy", "removed" or "migrated", or notes what a name used to be, and no test asserts that a deleted name is gone. Do not edit `vibe/` or `.vibelight/`. + + + + + +### Step 4: models.use with a handle, models.current, handle fields [completed] + +- Component: Fixed roles and Host map +- Tests: + - `cargo nextest run --locked -p runvil-model-client -p runvil-lua -p runvil-engine -p runvil -p harness-runner -p harness --all-features` + - `cargo fmt --all --check` + - `$env:CARGO_BUILD_WARNINGS = 'deny'; cargo clippy --locked -p runvil-model-client -p runvil-lua -p runvil-engine -p runvil --all-targets --all-features` + - `cargo +nightly-2026-09-05 xtask api --check` (after `--bless`) + - `rg -n "capabilities" crates/runvil-internal/model-client/src crates/runvil-internal/lua/src/models.rs crates/runvil-internal/lua/src/models-userdata.rs crates/runvil-internal/engine/src/execute/fill.rs` (a pass prints nothing and exits with code 1) + - `rg -n "with_capabilities" crates README.md prompts` (a pass prints nothing and exits with code 1) + - `git add -N .; git diff -U0 HEAD | rg -n -i "^\+.*(no longer|formerly|previously|legacy|deprecat|backward|replaces|migrat)"` (a pass prints nothing and exits with code 1) + +Finish the author-facing Lua surface for model roles: `models.current()`, `models.use` accepting a handle, and a handle without the keyword-set field. + +- Placement: last step of the "Host map to model set" piece. It adds behavior on top of the selection that step 3 made hold a whole binding, so it comes after it. It stays out of the front matter piece because nothing here reads the parser. +- Build state: the workspace builds and passes after this step. The facade listing changes (`ModelBinding::capabilities` and `with_capabilities` go), so regenerate `crates/runvil/public-api.txt` with `cargo +nightly-2026-09-05 xtask api --bless`. +- Behavior: + - `models.current()` returns the handle for the section's current selection, which starts on the `default` role. With an empty caller map it fails with an error containing `the caller supplied no model`. + - `models.use(role_or_handle, options?)` returns the handle. Given a handle (from `models.current()`, `models.get(...)`, or an earlier `models.use`, including a raw-id handle) it stores `options.apply(handle's binding)` as the selection, so options apply on top of the handle. Any other argument type fails with `models.use takes a model role name or a model handle, got number` (the Lua type name replaces `number`). + - The handle exposes `label`, `name`, `model_id`, `description`, `context`, `thinking` and `provider`, plus the methods `infer` and `loop`. The `capabilities` field goes. Leave the duplicate `name` and `label` fields and the handle methods as they are. + - A handle's description is the model info's description, because a role has no description of its own. +- Artifacts: + - `crates/runvil-internal/lua/src/models.rs`: `models.current` and the handle branch of `models.use`, using `is_handle` and `LuaModelHandle` from `models-userdata.rs`. + - `crates/runvil-internal/lua/src/models-userdata.rs`: delete the `capabilities` field and its method. + - `crates/runvil-internal/model-client/src/model/options.rs`: delete the `capabilities` field of `ModelBinding`, `ModelBinding::capabilities()` and `with_capabilities`, and fix the docs. Update `options-tests.rs` and `model/tests.rs`. + - `crates/runvil-internal/engine/src/execute/fill.rs`: stop passing the declared keywords and description into the binding. The binding's description is `info.description()`. + - Update the tests and fixtures that assert `capabilities` or a declared role's `description` to assert the model info's description instead, find them with `rg -n "capabilities" crates`. +- Test cases (`crates/runvil-internal/lua/src/models-tests.rs` and engine tests with the map `{ default: A, strong: B }`): + - `models.current()` returns a handle with label `default` before any `models.use`, and after `models.use('strong')` returns a handle with label `strong` and B's `model_id`; + - `local prev = models.current(); models.use('strong'); models.use(prev)` puts the section back on A, so the next `models.infer` request names A; + - `models.use(prev, { temperature = 0.2 })` applies the temperature on top of the handle's binding, so the next request names the handle's model and its `options.temperature()` is `Some` of 0.2; + - with a UI snapshot, a raw-id handle from `models.get('vendor/raw-model')` is accepted by `models.use`, and the next `models.infer` request names `vendor/raw-model`'s gateway model name; + - `models.use(5)` fails with `models.use takes a model role name or a model handle, got number`; + - with an empty caller map, `models.current()` fails with an error containing `the caller supplied no model`; + - a handle exposes `label`, `model_id`, `description`, `context`, `thinking` and `provider`, and its `infer` and `loop` methods keep working in the existing tests. +- Rules: nothing you write (code, comments, docs, error messages, test names) says "replaces", "formerly", "previously", "no longer", "legacy", "removed" or "migrated", or notes what a name used to be, and no test asserts that a deleted name is gone. Do not edit `vibe/` or `.vibelight/`. + + + + + +### Step 5: Requirements-only front matter, Workshop contract and UI, then the commit gate [completed] + +- Component: Fixed roles and Host map +- Tests: + - `cargo nextest run --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-features` + - `cargo nextest run --locked -p workshop-server -p workshop-server-api` + - `node --test crates/workshop/ui/test/run-api.mjs crates/workshop/ui/test/run-panel.mjs` + - `npm test --workspace ui --prefix crates/workshop` + - `npm run typecheck --workspace ui --prefix crates/workshop` + - `npm run build --workspace ui --prefix crates/workshop` + - `node --test crates/workshop/ui/test/docs-claims.mjs` + - `cargo fmt --all --check` + - `$env:CARGO_BUILD_WARNINGS = 'deny'; cargo clippy --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-targets --all-features` + - `$env:CARGO_BUILD_WARNINGS = 'deny'; cargo clippy --locked -p workshop-server -p workshop-server-api --all-targets` + - `$env:RUSTDOCFLAGS = '-D warnings'; cargo doc --locked --workspace --no-deps --all-features --exclude workshop --exclude workshop-server --exclude workshop-server-api` + - `cargo +nightly-2026-09-05 xtask api --check` (after `--bless`) + - `rg -n "keywords" crates/workshop/ui/src/services/run-api.ts crates/workshop/ui/src/parts/run crates/workshop/ui/test/run-api.mjs crates/workshop/ui/test/run-panel.mjs` (a pass prints nothing and exits with code 1) + - `rg -n "keywords:" crates README.md prompts --glob "!crates/workshop/ui/**"` (a pass prints nothing and exits with code 1) + - `rg -n "models\.default|models\.bind|default_alias|ModelView|ModelSetError|ModelBindings|model_bindings|ModelRoles|ModelRolesIter|ModelKeyword|keyword_name|keyword_wire|with_capabilities|HardKeyword|resolve_model_binding|trivial fill|bound model role" crates README.md prompts` (a pass prints nothing and exits with code 1) + - `rg -n "ModelRole|RoleRequirements|role_requirements|RequirementCheck::(Thinking|Unbound)|fn prepare|prepare\(self" crates/runvil/public-api.txt` (a pass prints lines for the new facade surface: the `ModelRole` enum with `ALL` and `as_str`, `RoleRequirements` with `min_context` and `thinking`, `Frontmatter::role_requirements`, the `Thinking` and `Unbound` checks, `UnmetRequirement::role` as `ModelRole`, and `RunContext::prepare` taking `BTreeMap`) + - `git add -N .; git diff -U0 HEAD | rg -n -i "^\+.*(no longer|formerly|previously|legacy|deprecat|backward|replaces|migrat)"` (a pass prints nothing and exits with code 1) + +Make `models:` front matter optional and limited to requirements on the five roles. Give the Engine, the Workshop server and the Workshop UI the new reader. This is the last step of the first component, so its tests include the component's whole exit gate for the `runvil` repository. + +- Placement: this piece, "Requirements-only front matter", comes after the Host map piece because the requirement checks now read the parser's new accessor and the fixtures already use role names. The parser, the engine reader, the Workshop server and the Workshop UI are one step. Removing the parser's old types stops the engine and the server from compiling until they read the new accessor, and once the server sends the new JSON the UI's reader returns `null` for every model row, so a commit that left the UI behind would leave the run panel without its model rows. The design document is step 8, written after all the code, so it describes the final surface. +- Build state: the workspace and the UI build and pass after this step, and the whole gate above passes. The facade listing changes, so regenerate `crates/runvil/public-api.txt` with `cargo +nightly-2026-09-05 xtask api --bless`. If a gate command fails because of an earlier step's leftover (a deleted name still present, a doc link, a listing difference), fix it here. +- Behavior: + - Front matter `models:` maps role names to `{ min_context, thinking }`. `min_context` is a positive token count and `thinking` is a boolean. Both are optional, and `fast: {}` is valid and adds nothing. A prompt with no `models:` key is valid. + - Parse errors: an unknown role name (`reviewer`) names the five roles `default`, `strong`, `small`, `fast` and `creative` (the serde unknown-variant error for `ModelRole`); an unknown field (`tier`) fails with ``unknown field `tier`, expected `min_context` or `thinking` ``; a duplicate role key fails; `min_context: 0` fails. + - `Frontmatter::role_requirements(role: ModelRole) -> Option<&RoleRequirements>` is the only accessor. It is `None` when the prompt does not declare the role, and `Some` with neither field set for `role: {}`. + - `thinking: true` requires a model that can think and turns thinking on for that role's requests. `thinking: false` requires a model that can run without thinking and turns it off. A role's requirements are checked against the model that fills the role, fallbacks included. + - The Workshop server sends each declared role as `{ label, min_context, thinking }`, where `min_context` is a number or `null` and `thinking` is a boolean or `null`. The run panel shows one read-only row per declared role: the role label, then its requirements as text, built from the parts that are set and joined with `, `: `min context N` when `min_context` is set, `thinking` when `thinking` is `true`, `no thinking` when it is `false`. A role with no requirements shows its label with empty details. +- Artifacts (parser, `crates/runvil-internal/parser/src`): + - `contract/models.rs`: rename the struct `ModelRole` to `RoleRequirements { min_context: Option, thinking: Option }` with `#[serde(deny_unknown_fields)]` and accessors `min_context()` and `thinking()`. Delete `ModelKeyword`, `ModelRoles`, `ModelRolesIter`, and the struct's `keywords` and `description`. The `models` key deserializes into a `BTreeMap` (`ModelRole` from `runvil_types::models`). Rewrite the module doc. + - `contract.rs`: make the shared contract-map visitor (`deserialize_contract_map` and its `ContractKeys`) generic over the key type. `args` keeps its alias-grammar string keys, `models` uses `ModelRole`, and the duplicate-key rejection stays shared. + - `frontmatter.rs`: replace `models()` with `role_requirements`. `detail.rs`: delete `keyword_name`. `lib.rs`: update the exports (`RoleRequirements` in, `ModelKeyword`, `ModelRole`, `ModelRoles`, `ModelRolesIter` out). Update the re-export in `crates/runvil-internal/engine/src/parser.rs`. + - `contract/tests.rs`: rewrite the model-role tests for the new surface. +- Artifacts (engine and facade): + - `crates/runvil-internal/engine/src/execute/fill.rs`: read each role's requirements with `prompt.frontmatter().role_requirements(role)`. `min_context` gives `ContextMinimum` as before. `thinking: Some(true)` against `ThinkingMode::Never` gives `Thinking` with required `thinking: true` and actual `Never`, and `thinking: Some(false)` against `Always` gives required `thinking: false` and actual `Always`. A declared `thinking` sets `ModelInvocation::new().with_thinking(requirements.thinking())` on that role's binding only. Delete the uses of `ModelKeyword` and `detail::keyword_name`. + - `crates/runvil/src/lib.rs` (the `prompt` module): export `RoleRequirements` and drop `ModelKeyword`, `ModelRole`, `ModelRoles`, `ModelRolesIter`. Regenerate `crates/runvil/public-api.txt`. Callers of `Frontmatter::models()` in `crates/runvil/tests/suite/prompt.rs` and the engine's `test_support` move to `role_requirements`. +- Artifacts (Workshop server, `crates/workshop/server/src/routes/prompts.rs` and `prompts-tests.rs`): `ModelDto` becomes `{ label: String, min_context: Option, thinking: Option }`, built for every role in `ModelRole::ALL` order that `role_requirements` returns `Some` for, with `label` set to `role.as_str()`. A missing value serializes as `null`. Delete `keyword_wire`. +- Artifacts (Workshop UI): + - `crates/workshop/ui/src/services/run-api.ts`: `RunContractModel` becomes `{ label: string; minContext: number | null; thinking: boolean | null }`. Its reader checks `label` is a string, `min_context` is `null` or a number, and `thinking` is `null` or a boolean, and returns `null` for any other shape as the current reader does. + - `crates/workshop/ui/src/parts/run/run-rows.ts`: build each model row's details from `minContext` and `thinking` as described above, and drop the keyword and description text. + - `crates/workshop/ui/test/run-api.mjs` and `crates/workshop/ui/test/run-panel.mjs`: rewrite the fixtures to `{ label: "strong", min_context: 32000, thinking: true }` and rename the test titled "a model role renders keywords and min_context read-only" to say it renders min_context and thinking read-only. +- Artifacts (fixtures and prompts): every `keywords: [thinking]` becomes `thinking: true` and `keywords: [no-thinking]` becomes `thinking: false`. Every remaining soft keyword and role `description:` is deleted. Find them with `rg -n "keywords:|description:" crates prompts README.md` and review each hit inside a `models:` block. Do this in `crates/runvil-internal/engine/src/execute/tests/suite/prepare.rs`, `crates/runvil/tests/suite/prepare.rs`, `prompts/*.md`, `crates/workshop/agents/agents/chat.md` and any other fixture the search finds. The UI files are covered by the Workshop UI artifacts above. +- Test cases: + - Parser: a prompt with no `models:` key has `role_requirements` of `None` for all five roles; each role declared as `{ min_context: 4096, thinking: true }` reads back both values; `models: { fast: {} }` parses and gives `Some` with neither value; `models: { reviewer: {} }` fails and the error text names all five roles; `models: { strong: { tier: 1 } }` fails with the unknown-field text above; a duplicate role key fails; `min_context: 0` fails; `args` keys are still checked against the alias grammar (keep the existing `args` tests). + - Engine: `models: { strong: { thinking: true } }` against `ThinkingMode::Never` and `models: { small: { thinking: false } }` against `ThinkingMode::Always` are each refused, with the entries and notice lines from step 2; `thinking: false` on `small` makes a `models.use('small')` round's request have `thinking()` of `Some(false)`, and `thinking: true` on `strong` gives `Some(true)` for `strong` rounds only; `min_context` on a role with no entry of its own is checked against the `default` model. + - Workshop server: the prompt-contract JSON for `models: { strong: { min_context: 200000, thinking: true }, fast: {} }` has `models` equal to `[{ "label": "strong", "min_context": 200000, "thinking": true }, { "label": "fast", "min_context": null, "thinking": null }]`, and a prompt with no `models:` has `models` equal to `[]`. + - Workshop UI: the reader accepts `{ label: "strong", min_context: 32000, thinking: true }` and gives `minContext` 32000 and `thinking` `true`; it accepts `{ label: "fast", min_context: null, thinking: null }` and gives `null` for both; it returns `null` for an entry whose `thinking` is a string; the run panel row for `{ label: "strong", min_context: 32000, thinking: true }` reads `min context 32000, thinking`, the row for `thinking: false` with no `min_context` reads `no thinking`, and a role with neither shows only its label. +- Rules: nothing you write (code, comments, docs, error messages, test names, prompts, fixtures) says "replaces", "formerly", "previously", "no longer", "legacy", "removed" or "migrated", or notes what a name used to be, and no test asserts that a deleted name is gone. Do not edit `vibe/` or `.vibelight/`. + + + + + +### Step 6: Chains own the current model and children inherit it, then the commit gate [completed] + +- Component: Chain-level current model +- Tests: + - `cargo nextest run --locked -p runvil-engine --all-features chain_model` + - `cargo fmt --all --check` + - `$env:CARGO_BUILD_WARNINGS = 'deny'; cargo clippy --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-targets --all-features` + - `$env:CARGO_BUILD_WARNINGS = 'deny'; cargo clippy --locked -p workshop-server -p workshop-server-api --all-targets` + - `cargo nextest run --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-features` + - `cargo nextest run --locked -p workshop-server -p workshop-server-api` + - `cargo +nightly-2026-09-05 xtask api --check` + - `rg -n "models\.default|models\.bind|default_alias|ModelView|ModelSetError|ModelBindings|model_bindings|ModelRoles|ModelRolesIter|ModelKeyword|keyword_name|keyword_wire|with_capabilities|HardKeyword|resolve_model_binding|trivial fill|bound model role" crates README.md prompts` (a pass prints nothing and exits with code 1) + - `git add -N .; git diff -U0 HEAD | rg -n -i "^\+.*(no longer|formerly|previously|legacy|deprecat|backward|replaces|migrat)"` (a pass prints nothing and exits with code 1) + +Make each chain own the selection that its sections' VMs share, so a switch lasts across the chain's sections, and start every chain from the right value: a child from a copy of its parent's current model taken at dispatch, and the root walk from the model the H1 pass ended on. This is the last code step of the second component, so its tests include the component's exit gate for the `runvil` repository. + +- Placement: this component comes second because it needs the immutable model set, the `default` baseline and the selection argument from the first component (step 3), and the third component needs the copy at dispatch that this step adds. It is one piece in one step: ownership and inheritance share the `ChainStart` field and the same tests, and a split would need an in-between commit where every child starts on `default`, which no behavior in the plan wants. +- Build state: the workspace builds and passes after this step, and the gate above passes. The facade listing does not change. +- Behavior: + - A chain is the engine's unit of sequential execution: the H1 pass, the root walk, a `call` child, or a spawned task (a fanout arm is a spawned task). A section VM is the Lua VM built for one section. + - Each chain holds one `Arc>>`. It is created when the chain starts, from the chain's starting value, and every section VM of that chain gets that same `Arc` at injection. So `models.use` in one section is still in effect in the next section of the same chain, and no two chains share a selection. + - The starting value is the `default` binding for the H1 pass and for the root walk of a prompt that has no H1 blocks. It is `None` when the caller's map was empty, because `ctx.default_binding()` is `None` then. + - A `call` child and a `tasks.spawn` task (so each fanout arm) start with a copy of the parent chain's current model, taken when the call or spawn is dispatched. A spawned task waits on the ready queue with no VM until the scheduler admits it, so a copy taken at admission would let a switch the parent makes after spawning change the task's model. Copying at dispatch keeps a run's result independent of when the scheduler admits a queued task. + - The root walk that follows H1 starts with the model the H1 pass ended on. This holds for the walk that starts when H1 ends normally and for the walk that starts when a jump leaves H1. + - A child's switch never reaches its parent or its siblings, because each chain's selection is its own and inheritance copies a value. + - The `sys.model` value a section reads at its first tool call is the chain's current model at that moment. +- Artifacts (`crates/runvil-internal/engine/src/execute`): + - `scheduler/chain.rs`: add `model: Option` to `ChainStart` (the starting value) and a `selection: Arc>>` field to `Chain`, created in `start_chain` from `start.model`. Add `Chain::current_model(&self) -> Result>`, which clones the value under the lock and returns an `Error::internal` with `model selection lock was poisoned` on a poisoned lock. Add a `model: Option` parameter to `start_root_chain` and pass it into the `ChainStart` it builds, and update the reason on its `#[expect(clippy::too_many_arguments)]` so it names the five values a root start forwards. + - `scheduler/dispatch.rs` (`prepare_call`, which builds the call child's `ChainStart`) and `scheduler/tasks-spawn.rs` (the spawn dispatch that builds the task's `ChainStart`): set `model` to `chain.current_model()?` on the parent chain, read at that moment and before `start_chain` runs. + - `scheduler/h1.rs`: `start_h1` passes `self.ctx.default_binding()` to `start_root_chain`. `end_h1` passes `chain.current_model()?` of the H1 chain to the root walk it starts. `end_h1_at_jump` calls `end_h1`, so this one site covers the jump too. `scheduler/walk.rs`: `start_root_walk` passes `self.ctx.default_binding()`. + - `scheduler/test_hooks.rs`: its two `ChainStart` sites (the wedge chain and the probe chain) set `model: self.ctx.default_binding()`. + - `section_vm.rs` and `section_context-construct.rs` (whichever builds the section VM): take the chain's selection `Arc` as a parameter and pass it to injection, in place of the fresh selection step 3 creates per section. The scheduler's callers pass `&chain.selection`. +- Test cases (new file `crates/runvil-internal/engine/src/execute/tests/scheduler/chain_model.rs`, wired into `scheduler.rs`; `Scenario` with the map `{ default: A, strong: B, fast: C }`, each case reading `models.current().label` or `model_id`): + - a prompt whose `## One` runs `models.use('strong')` and whose `## Two` returns `models.current().label` returns `strong`, and a `models.infer` in `## Two` makes a request that names B; + - a prompt that never calls `models.use` returns `default` from `models.current().label` in every section; + - the `sys.model` value read at the first tool call of `## Two` reflects the `strong` switch made in `## One`; + - a `call` child inherits the parent's `strong` switch, and a spawned task inherits it; + - a child's `models.use('fast')` does not reach the parent (the parent's next `models.current().label` is still `strong`) or a sibling task; + - a `models.use('strong')` made in H1 is the starting model of the root walk, both when H1 ends normally and when a jump leaves H1; + - a parent that spawns two tasks under `tasks.concurrency(1)`, then switches to `fast` before the second, queued task is admitted, leaves that task on the model the parent had when it spawned; + - all arms of one 2-argument `tasks.fanout` start on the same model, with the concurrency limit set to 1 so later arms are queued; + - audit every existing test that assumed a fresh selection per section: find them with `rg -n "models\.use" crates/runvil-internal/engine/src` and fix any test whose expectation depended on a switch ending with its section. +- Rules: nothing you write (code, comments, docs, error messages, test names) says "replaces", "formerly", "previously", "no longer", "legacy", "removed" or "migrated", or notes what a name used to be, and no test asserts that a deleted name is gone. Do not edit `vibe/` or `.vibelight/`. + + + + + +### Step 7: Model-first tasks.fanout, then the commit gate [completed] + +- Component: Model-first fanout +- Tests: + - `cargo nextest run --locked -p runvil-engine -p runvil-lua --all-features fanout_model` + - `cargo fmt --all --check` + - `$env:CARGO_BUILD_WARNINGS = 'deny'; cargo clippy --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-targets --all-features` + - `$env:CARGO_BUILD_WARNINGS = 'deny'; cargo clippy --locked -p workshop-server -p workshop-server-api --all-targets` + - `cargo nextest run --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-features` + - `cargo nextest run --locked -p workshop-server -p workshop-server-api` + - `cargo +nightly-2026-09-05 xtask api --check` + - `rg -n "models\.default|models\.bind|default_alias|ModelView|ModelSetError|ModelBindings|model_bindings|ModelRoles|ModelRolesIter|ModelKeyword|keyword_name|keyword_wire|with_capabilities|HardKeyword|resolve_model_binding|trivial fill|bound model role" crates README.md prompts` (a pass prints nothing and exits with code 1) + - `git add -N .; git diff -U0 HEAD | rg -n -i "^\+.*(no longer|formerly|previously|legacy|deprecat|backward|replaces|migrat)"` (a pass prints nothing and exits with code 1) + +Add the model-first form `tasks.fanout(model, worker, collection)`, implemented entirely in the Lua fanout shim. This is the third component's only code step, so its tests include the component's exit gate for the `runvil` repository. + +- Placement: this component is last because the shim relies on the second: spawned arms copy the parent's selection at dispatch, so setting the parent's selection before spawning sets every arm's model. It is a single piece with a single code step, because the shim, its privileged captures and its tests are one behavior that one set of tests covers. +- Build state: the workspace builds and passes after this step, and the gate above passes. The scheduler, the spawn request and every protocol type stay as they are. +- Behavior: + - `tasks.fanout(worker, collection)` is unchanged: arms inherit the chain's current model. + - `tasks.fanout(model, worker, collection)` takes a model handle first. The shim reads the parent's current selection, sets the selection to the handle's binding, spawns every arm, and restores the saved selection before the first join and also before it re-raises a spawn failure. The parent's model after the call is the model it had before. No author code runs in the parent between the switch and the restore, and `models.use` emits no event, so the switch cannot be observed. + - Inside an arm, `models.use('strong')` affects only that arm and its own children, and `models.use('default')` selects the `default` role, not the model the fanout was given. + - Errors: a first argument that is not a model handle (including `nil`) fails with `tasks.fanout: the first argument must be a model handle, got nil` (the Lua type name replaces `nil`), and any argument count other than 2 or 3 fails with `tasks.fanout takes (worker, collection) or (model, worker, collection)`. +- Artifacts (`crates/runvil-internal/lua/src`): + - `__impl_fanout.lua`: handle the 3-argument form with the switch, spawn-all and restore above. Use the privileged captures the shim receives at install: `get_selection()` returns the chain's current handle or `nil`, `set_selection(handle_or_nil)` sets the selection (`nil` clears it) and fails for any other value, and `is_model_handle(value)`. Reject a non-handle first argument before switching. + - `coro.rs`: install the three captures when the shim is installed, over the selection `Arc>>` that injection received (the calling chain's own since step 6) and `LuaModelHandle`. They are passed to the shim chunk and are not reachable from author code. +- Test cases (new file `crates/runvil-internal/engine/src/execute/tests/scheduler/fanout_model.rs`, wired into `scheduler.rs`; `Scenario` with the map `{ default: A, strong: B, fast: C }`, and Lua unit tests in `crates/runvil-internal/lua/src` for the captures): + - a 3-argument fanout with `models.get('fast')` runs every arm on C: each arm returns `models.current().model_id` and every result is C's; + - after the call the parent's model is what it was before: a parent on `strong` reads `strong` from `models.current().label` after the fanout; + - a spawn failure part-way still restores the parent's model: trigger the depth-cap refusal the shim already re-raises (see the comment above `fanout` in `__impl_fanout.lua`), catch it with `pcall`, and check that `models.current().label` is the label before the call; + - a raw-id handle works with an empty caller map: with a UI snapshot set and no models in the map, `tasks.fanout(models.get('vendor/raw-model'), worker, items)` runs every arm on that gateway model; + - a first argument that is not a handle (a string, a number, and `nil`) fails with the text above, and an argument count other than 2 or 3 fails with the second error text; + - `models.use('default')` inside an arm selects the `default` role, and `models.use('strong')` inside an arm leaves the parent's and the other arms' models alone; + - the 2-argument form still gives every arm the parent's current model. +- Rules: nothing you write (code, comments, docs, error messages, test names) says "replaces", "formerly", "previously", "no longer", "legacy", "removed" or "migrated", or notes what a name used to be, and no test asserts that a deleted name is gone. Do not edit `vibe/` or `.vibelight/`. + + + + + +### Step 8: Design document for roles, the chain's current model and fanout [completed] + +- Component: Model-first fanout +- Tests: + - Run from the `runvil-design` repository, the sibling of `runvil`: `rg -n "models\.default|models\.bind|default_alias|ModelView|ModelSetError|ModelBindings|model_bindings|ModelRoles|ModelRolesIter|ModelKeyword|keyword_name|keyword_wire|with_capabilities|HardKeyword|resolve_model_binding|trivial fill|bound model role" design` (a pass prints nothing and exits with code 1) + - Run from the `runvil-design` repository: `git add -N .; git diff -U0 HEAD | rg -n -i "^\+.*(no longer|formerly|previously|legacy|deprecat|backward|replaces|migrat)"` (a pass prints nothing and exits with code 1) + +Rewrite the models part and the task descriptions of `design/architect-runvil-prompting-language.md` in the `runvil-design` repository to describe the behavior steps 1 to 7 ship. This step's commit goes in `runvil-design`, not in `runvil`. + +- Placement: the last step of the plan. It is written once, after all the code, so the document describes the finished surface of the three components and not three partial states. It is its own step because it is a commit in a different repository. It is filed under the third component, which finished its code at step 7, so the plan keeps three components. +- Build state: documentation only. Nothing in `runvil` changes. +- Behavior to document (present tense, no history): + - Every prompt has exactly five roles: `default`, `strong`, `small`, `fast`, `creative`. There are no custom roles. Front matter `models:` is optional and holds only requirements per role, `min_context` (a positive token count) and `thinking` (`true` requires a model that can think and turns thinking on for that role's requests, `false` requires one that can run without thinking and turns it off). An unknown role name or field is a parse error. + - The Host passes a role-to-model map to `prepare`. A role with no entry uses the `default` entry. An empty map means the Host has no model, and a prompt fails only when it calls a model. A non-empty map without `default` is refused before the run starts. Requirements are checked against the model that fills the role, fallbacks included. + - The Lua surface: a chain starts on the `default` role. `models.use(role_or_handle, options?)` switches the chain's current model and returns the handle, and the switch lasts across that chain's sections until the next switch. `models.current()` returns the current handle, `models.get(role)` inspects without switching, and `models.infer` and `models.loop` run on the current model or an explicit handle. A handle exposes `label`, `model_id`, `description`, `context`, `thinking` and `provider`, plus `infer` and `loop`. Error texts: `models.use label "reviewer" is not a model role: expected default, strong, small, fast or creative`, and `the caller supplied no model`. + - `call` children, `tasks.spawn` tasks and fanout arms start with a copy of the parent's current model, taken when the call or spawn is dispatched. The root walk starts with the model the H1 pass ended on. A child's switch never reaches its parent or its siblings. + - The save-and-restore pattern is `local prev = models.current(); models.use('creative'); ...; models.use(prev)`. A `models.use('strong')` for one expensive step leaves every later section of that chain on `strong`, which is a cost to plan for, and the save-and-restore pattern undoes it. + - `tasks.fanout(worker, items)` runs every arm on the chain's current model. `tasks.fanout(model, worker, items)` takes a model handle first and runs every arm on it, without switching the parent: the parent's model after the call is what it was before, including when a spawn fails part-way. Example: `tasks.fanout(models.get('fast'), worker, items)`. Inside an arm, `models.use('strong')` affects only that arm and its children, and `models.use('default')` selects the `default` role. Error texts: `tasks.fanout: the first argument must be a model handle, got nil` and `tasks.fanout takes (worker, collection) or (model, worker, collection)`. The save-and-restore example with a 2-argument fanout stays valid: `local prev = models.current(); models.use('creative'); local drafts = tasks.fanout(worker, items); models.use(prev)`. + - Wherever the document says a switch lasts for the rest of the section, it says the chain instead. +- Artifacts: in `design/architect-runvil-prompting-language.md`, rewrite the `## Models` section, every other mention of `models.bind` and `models.default`, and the `tasks.fanout` and `tasks.spawn` descriptions and examples. Find the places with `rg -n "models\.bind|models\.default|keywords|models\.use|models\.current|selection|tasks\.fanout|tasks\.spawn" design`. The `models.bind` and `models.default` hits are in the deferred-work list near the top, the sample prompt that calls `models.default("writer")`, the `models.bind` and `models.default` headings and effects, the paragraph about the section's model selection and explicit handles, and the later examples. Rewrite each example in place to run on the `default` role or to call `models.use` with a role. Follow the document's existing heading and example style. +- Rules: nothing you write says "replaces", "formerly", "previously", "no longer", "legacy", "removed" or "migrated", or notes what a name used to be. Do not edit dated logs. + + + + diff --git a/vibe/2026-10-19-2-pack-plugin-rename.md b/vibe/2026-10/2026-10-19-2-pack-plugin-rename.md similarity index 100% rename from vibe/2026-10-19-2-pack-plugin-rename.md rename to vibe/2026-10/2026-10-19-2-pack-plugin-rename.md diff --git a/vibe/2026-10-19-3-pack-rename-debt.md b/vibe/2026-10/2026-10-19-3-pack-rename-debt.md similarity index 100% rename from vibe/2026-10-19-3-pack-rename-debt.md rename to vibe/2026-10/2026-10-19-3-pack-rename-debt.md diff --git a/vibe/2026-10-20-1-library-plugin-loading.md b/vibe/2026-10/2026-10-20-1-library-plugin-loading.md similarity index 100% rename from vibe/2026-10-20-1-library-plugin-loading.md rename to vibe/2026-10/2026-10-20-1-library-plugin-loading.md diff --git a/vibe/2026-10-20-1-remove-allow-tasks.md b/vibe/2026-10/2026-10-20-1-remove-allow-tasks.md similarity index 100% rename from vibe/2026-10-20-1-remove-allow-tasks.md rename to vibe/2026-10/2026-10-20-1-remove-allow-tasks.md diff --git a/vibe/2026-10-20-2-plugin-loading-debt-removal.md b/vibe/2026-10/2026-10-20-2-plugin-loading-debt-removal.md similarity index 100% rename from vibe/2026-10-20-2-plugin-loading-debt-removal.md rename to vibe/2026-10/2026-10-20-2-plugin-loading-debt-removal.md diff --git a/vibe/2026-10-21-1-mcp-script-raw-text.md b/vibe/2026-10/2026-10-21-1-mcp-script-raw-text.md similarity index 100% rename from vibe/2026-10-21-1-mcp-script-raw-text.md rename to vibe/2026-10/2026-10-21-1-mcp-script-raw-text.md diff --git a/vibe/2026-10-21-1-section-messages.md b/vibe/2026-10/2026-10-21-1-section-messages.md similarity index 100% rename from vibe/2026-10-21-1-section-messages.md rename to vibe/2026-10/2026-10-21-1-section-messages.md diff --git a/vibe/2026-10-22-1-section-messages-debt-removal.md b/vibe/2026-10/2026-10-22-1-section-messages-debt-removal.md similarity index 100% rename from vibe/2026-10-22-1-section-messages-debt-removal.md rename to vibe/2026-10/2026-10-22-1-section-messages-debt-removal.md diff --git a/vibe/2026-10-22-2-utf8-library.md b/vibe/2026-10/2026-10-22-2-utf8-library.md similarity index 100% rename from vibe/2026-10-22-2-utf8-library.md rename to vibe/2026-10/2026-10-22-2-utf8-library.md diff --git a/vibe/2026-10-23-1-one-offer-api.md b/vibe/2026-10/2026-10-23-1-one-offer-api.md similarity index 100% rename from vibe/2026-10-23-1-one-offer-api.md rename to vibe/2026-10/2026-10-23-1-one-offer-api.md diff --git a/vibe/2026-10-24-1-one-offer-api-debt.md b/vibe/2026-10/2026-10-24-1-one-offer-api-debt.md similarity index 100% rename from vibe/2026-10-24-1-one-offer-api-debt.md rename to vibe/2026-10/2026-10-24-1-one-offer-api-debt.md diff --git a/vibe/2026-10-25-1-tools-kernel-redo.md b/vibe/2026-10/2026-10-25-1-tools-kernel-redo.md similarity index 100% rename from vibe/2026-10-25-1-tools-kernel-redo.md rename to vibe/2026-10/2026-10-25-1-tools-kernel-redo.md diff --git a/vibe/2026-10-26-1-engine-slop-cleanup.md b/vibe/2026-10/2026-10-26-1-engine-slop-cleanup.md similarity index 100% rename from vibe/2026-10-26-1-engine-slop-cleanup.md rename to vibe/2026-10/2026-10-26-1-engine-slop-cleanup.md diff --git a/vibe/2026-10-26-2-engine-scheduler-split.md b/vibe/2026-10/2026-10-26-2-engine-scheduler-split.md similarity index 100% rename from vibe/2026-10-26-2-engine-scheduler-split.md rename to vibe/2026-10/2026-10-26-2-engine-scheduler-split.md diff --git a/vibe/2026-10-27-1-pr172-debt-removal.md b/vibe/2026-10/2026-10-27-1-pr172-debt-removal.md similarity index 100% rename from vibe/2026-10-27-1-pr172-debt-removal.md rename to vibe/2026-10/2026-10-27-1-pr172-debt-removal.md diff --git a/vibe/2026-10-27-2-plugin-std-file-tools.md b/vibe/2026-10/2026-10-27-2-plugin-std-file-tools.md similarity index 100% rename from vibe/2026-10-27-2-plugin-std-file-tools.md rename to vibe/2026-10/2026-10-27-2-plugin-std-file-tools.md diff --git a/vibe/2026-10-29-1-lua-slop-cleanup.md b/vibe/2026-10/2026-10-29-1-lua-slop-cleanup.md similarity index 100% rename from vibe/2026-10-29-1-lua-slop-cleanup.md rename to vibe/2026-10/2026-10-29-1-lua-slop-cleanup.md diff --git a/vibe/2026-10-29-2-model-client-slop.md b/vibe/2026-10/2026-10-29-2-model-client-slop.md similarity index 100% rename from vibe/2026-10-29-2-model-client-slop.md rename to vibe/2026-10/2026-10-29-2-model-client-slop.md diff --git a/vibe/2026-10-30-1-engine-vibelight-cleanup.md b/vibe/2026-10/2026-10-30-1-engine-vibelight-cleanup.md similarity index 100% rename from vibe/2026-10-30-1-engine-vibelight-cleanup.md rename to vibe/2026-10/2026-10-30-1-engine-vibelight-cleanup.md diff --git a/vibe/2026-11-01-1-std-read-listdir-grep.md b/vibe/2026-11-01-1-std-read-listdir-grep.md new file mode 100644 index 000000000..fca265f5d --- /dev/null +++ b/vibe/2026-11-01-1-std-read-listdir-grep.md @@ -0,0 +1,590 @@ +--- +name: std Read, ListDir, and Grep +overview: Make std Read and Grep real and add a seventh tool, std/dir (wire name ListDir), so a model can read, search, and explore directories with no shell. Write, StrReplace, Delete, and Glob stay stubs. Read needs one new Access method, so this plan also changes the VFS. +todos: + - id: vfs-read-range + content: "Add Access::read_range in runvil-vfs (gate, claim_read, fire event, delegate to the existing VfsAccess::read_range) with VFS tests and a regenerated public-api.txt, in its own commit" + status: completed + - id: read-tool + content: "Add the shared plugin-std code (regex workspace dependency, refusal text, argument helpers, plural, text module) and implement Read in src/read/ with the new read.json; dispatch Read and flip its integration tests" + status: completed + - id: dir-tool + content: "Add the glob matcher and implement ListDir in src/dir/ with the new dir.json; embed it in spec.rs, move the tool-count tests from six to seven, dispatch ListDir, and flip its integration tests" + status: completed + - id: grep-tool + content: "Change schemas/grep.json to the bundle schema plus literal, implement Grep in src/grep/ with its yield test, dispatch Grep, flip its integration tests, and finish with the full gate set" + status: completed +isProject: false +--- + +# std Read, ListDir, and Grep + + + +## Product Requirements + +The `std` Pack (crate `plugin-std`) ships six file tools that are stubs returning one fixed not-implemented text. This work makes Read and Grep real and adds a seventh tool, ListDir, so a model can explore directories, read files, and search with no shell. Every tool works only through the run's virtual file system, so the run's Mode and policy gate each access. Write, StrReplace, Delete, and Glob stay stubs. Until a shared ignore module exists, the walks skip only `.git`. + +- Problem and users: + - Today all six `std` tools return the same fixed text (`crates/plugin-std/src/pack.rs`, stub added in commit `08801150f`), so a model in a Runvil run cannot read, search, or explore files. + - Users are models in Runvil runs, Lua authors who offer the tools by Lua id (`std/read`, `std/dir`, `std/grep`), and the maintainer. +- Goals: + - Read returns line-numbered windows of text files at bounded cost. + - ListDir (Lua id `std/dir`) returns a bounded directory tree. + - Grep searches file contents with Rust regex syntax, stays cancellable, and pages its results. + - Every failure names the path and says what to do next. +- Non-goals: + - Real Write, StrReplace, Delete, and Glob. + - A shell Pack, user approval, PDFs, and images. + - The read record that Write will need, `.gitignore` pruning, and a VFS-level search. + - Mounting a real directory in Workshop, which mounts empty in-memory storage today (`VfsRef::default()`). +- Success criteria: + - Read, ListDir, and Grep return the texts in the Functional Specification against the in-memory backend and against a real directory, and the four remaining stubs still return the stub text. + - Every exit criterion in the Testing Plan passes. +- Constraints: + - Model-facing (wire) names copy Cursor's, except ListDir, whose Cursor counterpart `LS` is deprecated and gated behind a setting. Lua ids are `std/`. + - Text only, and no dotfile filtering. + - The Engine and the Harness do not change, and the VFS changes only through `Access::read_range`. + - A Plugin crate may name only `runvil-plugin`, `shared-*` crates, and outside libraries (`crates/build-xtask/src/product.rs`), so `plugin-std` never depends on the vfs crate. + - No Rust file passes 500 lines. + - No existing file under `vibe/` is edited, because those plans are historical. The run's own copy of this plan and `vibe/ACTIVE` are the only files it adds or changes there. +- Open questions: None + +## Functional Specification + +Read, ListDir, and Grep take JSON arguments and return text. Read pages through one file by line window, ListDir lists a tree breadth first under a byte cap, and Grep walks a tree in path order and pages its matches. Each refuses bad arguments by name, names the path in every failure, and returns file-derived text as untrusted. No cut ever splits a UTF-8 code point. + +- Actors and workflows: + - A model calls a tool by wire name (`Read`, `ListDir`, `Grep`), and a Lua script can offer or call it by Lua id. + - The Pack validates the arguments, reads through the call's `Access`, and returns text. + - A typical flow is ListDir to find a directory, Grep to locate text, Read to open a window of a file, then Read again with the offset from the trailer. +- Inputs and outputs: + - Read (`std/read`): + - Arguments are `path` (string, required), `offset` (integer), and `limit` (integer). `limit` below 1 is refused with ``std_Read: `limit` must be at least 1, and the call passed 0``. + - The tool calls `Access::stat` first and refuses a directory, FIFO, socket, or device by name. It then reads at most 16,777,217 bytes. If more than 16,777,216 bytes come back the answer is `File is too large to read: {path} is over the limit of 16777216 bytes.` Read never truncates a file that is over the ceiling. + - Binary check: if any of the first 8,192 bytes is NUL, or at least 5% of them are control bytes other than tab, LF, and CR, the answer is `File content is binary data, not text: {path}`. There is no extension list. + - Decoding drops a UTF-8 mark, decodes UTF-16 by its mark, and otherwise decodes UTF-8 lossily. Lines split on `\n`, the empty element after a final newline is dropped, and one `\r` is stripped from each line, so `a\nb\n` and `a\nb` both have two lines and a zero-byte file has none. A file with no lines answers `File is empty.` as a success. + - Window: the default offset is 1, offset `0` reads as 1, and a negative offset counts from the end (`-1` is the last line, and with no limit it reads the last `|offset|` lines). An overlapping window is clamped. A window that selects nothing fails with `Offset 11 is beyond the end of /notes.md, which has 10 lines.` + - Numbering: every line gets a six-wide right-aligned number, a pipe, and no space (` 1|text`), always on, widening past six digits. + - Cap: the byte cap is 100,000 bytes of numbered lines, counting each prefix and newline and not the trailer. Read stops before the line that would pass it and ends a cut read with `[Showing lines 1-2083 of 5000 (100000-byte limit). Use offset=2084 to continue.]`. The parenthesis is dropped when the caller's `limit` ended the read, and the last page prints no trailer. + - First-line cut: a first line that alone passes the cap is clipped to the longest prefix that fits the 99,992-byte allowance (the cap less the 7-byte prefix and the newline) and ends on a character boundary. The notice reports the bytes kept, for example `[Line 1 is 150000 bytes and is cut after 99992 bytes. ...]` for ASCII and `cut after 99990 bytes` for 3-byte characters. That is the only cut inside a line. UTF-16 is decoded in full before the cut. + - Notes: a UTF-16 file adds `[Note: the file is UTF-16LE and was decoded to text.]`, and replaced bytes add `[Note: n invalid UTF-8 sequences in this file were shown as U+FFFD.]`. + - Conflict: if another call in the same round holds a write claim on the path, the answer is `{path} was changed, or is being changed, by another call in this round, so it cannot be read in this round. Call Read again in your next message.` Read never retries inside the call. + - Description in `read.json`: remove the `cat -n` and 2,000-character-line text. State the six-wide format and that the prefix is not part of the line, the 100,000-byte cap and trailer, the 16777216-byte ceiling, that binary files are refused, that Read opens any path the call names, and that a link is read through unless it leaves the run's filesystem. The description is built from the same constants the code uses. + - ListDir (`std/dir`, new `schemas/dir.json`): + - Arguments are `target_directory` (string, required) and `ignore_globs` (array of strings). A blank `target_directory` is refused before any I/O. + - Output is a tree: the root line is the absolute virtual path plus `/`, then a two-space indent per depth, `- ` bullets, and a trailing `/` on directories only. Files and directories share one list in byte order of the name with `/` appended to a directory's name. An empty directory prints `... no children found ...`, and a directory shown but not expanded ends in `...`. + - The walk is breadth first, so it stops at the byte cap instead of walking a whole tree first. Links are listed by name and never entered. Entries named `.git` are skipped, and every other dotfile is listed. + - The body cap is the shared 15,000 bytes. Over budget the listing ends with ` ...` and `target_directory is too large (either too many children or their names are too long). Immediate children were truncated. Try a more specific target_directory or ignore_globs.` + - `ignore_globs` use the shared glob matcher, so a pattern with no leading `/` matches at any depth. The description says so. + - Failures: `Path does not exist: {path}` and `Path is not a directory: {path}`. + - Dropped from Cursor's `LS`: extension-count summaries, the 5-second timeout, the terminal-log special case, and the `Error listing directory:` prefix. + - Grep (`std/grep`): + - Parameters, in this order: `pattern`, `path`, `glob`, `output_mode`, `-B`, `-A`, `-C`, `-i`, `type`, `head_limit`, `offset`, `multiline`, then `literal` (boolean, default false). `-n` and `context` are deleted, and `-C` takes the text that was on `context`. `output_mode` defaults to `content`. `head_limit` has no default, a minimum of 0, and 0 means no limit. `-n: true` is accepted and `-n: false` is refused with ``std_Grep: `-n` must be true or left out, because line numbers are always shown.`` Any other key is refused by name with no aliases. + - Dialect: the Rust `regex` crate, compiled before the walk, with `-i` as case-insensitive, multi-line anchors always on, and dot-matches-newline and CRLF handling only with `multiline`. Outside multiline mode matching is line by line, and in it the whole text is one subject. `literal` matches the escaped pattern text. A bad pattern fails with ``std_Grep: Invalid regular expression "": . Grep matches with Rust regex syntax, which has no look-around and no back-references. To match the text literally, set literal to true.`` + - Search root: `path` is statted first. A directory is walked and a file is searched alone. + - Order: each directory's entries are visited in byte order of the name with `/` appended to a directory's name, which is the same order as `Access::glob`. For example `/B`, `/a-b`, `/a.txt`, `/a/x`, `/b`. No clock is read. + - Limits: stop at 100,000 files considered or 1 GiB read. A single file over 16 MiB is a counted skip, decided from the size in the directory listing before any read. + - Skips: no single entry fails the search. Each skip is counted by reason (binary, unreadable directory, unreadable file, over 16 MiB, symbolic link, name with a control character, another call in this round, denied by policy), the first five are named, and a zero-match result says how many files were searched: `No matches found. Searched 412 files.` + - Counting: `head_limit` and `offset` are global across files and counted in entries (a matching line, a file, or a count row), and context lines are never entries. `head_limit` rounds up and `offset` rounds down. The search collects one extra entry to decide whether a notice is due, so an exact fit prints none. + - Paging: there is no cache, and each call re-runs from the start. A cut ends with `[Showing 1-50 of at least 51 matching lines. Use offset=50 to see more, or narrow the search.]`, the last page shows an exact total, and an offset past the end says `No more results at offset=N.` A stop caused by a limit offers no `offset`. + - Output cap: 15,000 bytes of body in every mode, cut on a line boundary, with notice lines not counted. A matching line over 1,000 bytes becomes a window of at most 1,000 bytes that contains the first match and starts up to 200 bytes before it, with `[... N bytes omitted]` markers. A context line over 1,000 bytes is clipped the same way. + - Render: there is no envelope. Each file's absolute virtual path appears once, then ` N:text` for a match and ` N-text` for context, a blank line between files and none at the end, and a gap line (two spaces, then two hyphens) between non-adjacent groups whenever a context flag is above 0. `-A` and `-B` each override their side after `-C` sets both, and overlapping context merges. `files_with_matches` prints one path per line, and `count` prints `path:count`. + - Text: files are read with `Access::read` and decoded through the shared text rules, so a Grep line number names the line Read prints. Binary files are skipped in every output mode with the `binary` reason, so one file never appears in one mode and not in another. Replaced bytes add `[Note: invalid UTF-8 in 2 files was shown as U+FFFD: /a.txt, /b.txt]`. + - Description in `grep.json`: names the dialect, order, limits, skips, and the ignore rule. It names `Rust regex`, `literal`, `15000`, `16777216`, and `skipped`, and does not name `ripgrep`, `rg `, `several thousand`, or `Shell`. + - `type` uses a built-in table of about fifteen common names. + +- States and validation: + - Every argument error has the shape ``std_{Wire}: `{name}` {problem}``. `null` reads as absent, and numeric strings are not coerced. In Grep an empty `path`, `glob`, or `type` reads as absent. + - Undeclared keys are refused by name with the tool's parameter list, because the shared argument check lets them through (`crates/plugin-std/src/check.rs`). An empty or whitespace-only `path` or `target_directory` is refused before any I/O. + - Grep builds its argument errors in this order: empty pattern (``Pass "." to match every line``), negative `-A`, `-B`, or `-C`, negative `head_limit` or `offset`, unknown `type` (``Unrecognized file type "foo". Use a type such as js, ts, py, rust, go or java, or filter with glob.``), invalid `glob`, the pattern compile, then the `path` stat. + - Glob grammar for Grep's `glob` and ListDir's `ignore_globs`: only `*` and `**` are special. `*` matches zero or more bytes except `/` inside one segment and does match dotfiles. `**` as a whole segment matches zero or more whole segments. Every other byte is literal, including `?`, `[`, `{`, `!`, and `.`, and matching is case-sensitive. A pattern with no leading `/` matches at any depth, which differs from `Access::glob`, where a relative pattern is anchored at the access root (`crates/runvil-internal/vfs/src/glob.rs`). A pattern is refused when it is empty, has more than two `*` in a row, uses `**` as less than a whole segment, contains a backslash or a control byte, or is longer than 1,024 bytes, which are the patterns `Access::glob` refuses. +- Errors and recovery: + - One shared function turns every `VfsError` into one model-readable sentence, so one variant reads the same in every tool. Texts it owns: `File not found: {path}`, `Path is a directory, not a file: {path}`, `Path is neither a file nor a directory: {path}`, `Path does not exist: {path}` for a missing search root, the conflict sentence, and `I/O failure on {path} ({kind}): {message}`. + - A message echoes the path as the model wrote it, never the canonical path, and names no other tool. `PermissionDenied` and `ApprovalRequired` reasons pass through unchanged, and a `Display` fallback keeps a variant added later readable. + - Recovery is in the text: Read's trailer names the next offset, Grep's footer names the next offset or says to narrow the search, and the conflict sentence says to call Read again in the next message. + - A cancel or stop drops a running call at its next await point. Grep yields so a long search can be dropped promptly. +- Security and privacy behavior: + - Text derived from files is returned as untrusted output. The one exception is Grep's bare `No matches found` line, which may be trusted. + - Every access goes through the call's `Access`, so the run's Mode and policy gate it. The Pack never touches the real filesystem directly. + - Read opens any path the call names inside what the run has mounted. ListDir and Grep skip entries named `.git` and apply no other ignore rule, and no tool filters dotfiles. +- Acceptance criteria: + - Each text in this section is returned byte for byte, and the four remaining stubs return the stub text unchanged. + - Against a real directory, a model can list it with ListDir, find text with Grep, and read a window of the file with Read, then read the next window using the offset from the trailer. + - No Read, ListDir, or Grep output contains a split UTF-8 code point at any cut, for any cap value. + + + + +## Technical Design + +The Pack stays a thin layer over the call's `Access`: each tool validates its arguments, reads through the VFS, and returns text. Three shared modules (refusal text, text decoding, and the glob matcher) plus the argument helpers keep the three tools consistent. The VFS gains `Access::read_range`, which changes the public API file. Grep yields cooperatively so a running search can be cancelled. Plugin tests cannot choose their own `VfsRef`, so tests lean on `TestCall`'s in-memory storage. + +- Architecture: + - `Pack::call` returns a boxed future that the Harness polls inside its effect loop (`crates/runvil-plugin/src/pack.rs`, `crates/harness-internal/runner/src/host-run.rs`). A future that wakes its own task and returns `Pending` once hands control back to the Host's executor before the call completes. A cancel or stop drops the future, and nothing applies a time limit to a running call (`crates/harness-internal/runner/src/effect_loop.rs`). + - Grep yields after every 32 files or 2 MiB read, through a small hand-written future that wakes its own task and returns `Pending` once. Read and ListDir do not yield, because the 16 MiB ceiling and the 15,000-byte cap bound their work. + - Grep is a pure `find(access, request)` that returns what it found, and a separate `render` that turns that into text. It walks with `Access::list` and reads with `Access::read`. There is no VFS-level search. + - `Access::list` promises no order. The stock backends sort by name, but the trait does not say so (`crates/runvil-internal/vfs/src/traits/backend.rs`), so the Pack sorts each listing itself by name with `/` appended to a directory's name. That order equals the order of `Access::glob`, which sorts full path strings bytewise. +- Modules and interfaces: + - Each tool's source lives in a directory named after its Lua id: `src/read/`, `src/dir/`, and `src/grep/`. The entry file sits beside its directory (`src/read.rs` next to `src/read/`), and there is no `mod.rs`. A tool's directory never reaches into another tool's directory. + - Write, StrReplace, Delete, and Glob keep their stub in `src/pack.rs` and get a directory when they get real code. + - Shared code stays in `src/`: `refusal.rs` (one function from a tool name, the path as written, and a `VfsError` to a sentence), `text.rs`, and `matcher.rs`, plus the argument helpers in `check.rs`. + - The text module has `decode`, `sniff_binary`, `split_lines`, and `clip_to_boundary`. `clip_to_boundary` is the only function that shortens text inside a line. It takes a `&str`, returns a `&str`, steps to the nearest character boundary with `str::is_char_boundary`, and is never bypassed by slicing on a raw byte index. + - The argument helpers are `refuse_unknown` (builds the undeclared-key error), a blank-path helper, and `plural(n, noun)` for every count in a message. + - Caps and limits (the 100,000-byte cap, the file, byte, and size limits, the 32-file slice) are parameters of the rendering and search functions with the constants as defaults, so tests can lower them. + - A module's unit tests are `tests.rs` inside its own directory. The existing `pack-tests.rs`, `spec-tests.rs`, and `check-tests.rs` siblings stay as they are, and no new `#[path]` attribute is added for layout. +- File and public API changes: + - `Access::read_range(path, offset, len)` is added in `crates/runvil-internal/vfs`. It takes the same four steps as `read`: gate by policy, admit a read claim, fire the read event, and delegate to the backend's existing `VfsAccess::read_range` (`crates/runvil-internal/vfs/src/traits/backend.rs`). `RealBackend` overrides that method with a bounded read. The trait's default reads the whole file and slices it, so a backend that does not override it is not bounded. + - The `Access::read_range` addition is its own commit with a VFS test and a regenerated `crates/runvil/public-api.txt`. + - If a new `Access` method is refused, the fallback is a `stat` ceiling for regular files kept in the Pack, which accepts that a link to a huge file looks small. Confidence: medium. + - Schemas in `crates/plugin-std/schemas/`: `read.json` (integer `offset` and `limit`, new description), `grep.json` (new parameters and description), and a new `dir.json`. Each change is recorded in `schemas/EDITS.md`. `spec.rs` embeds seven files, and the tool-count tests move from six to seven. + - `plugin-std` adds `regex` from the workspace (`regex = "1"` in the root `Cargo.toml`). `globset`, `sha2`, and `ignore` are not added. +- Data, persistence, failure, security, and privacy constraints: + - Nothing persists. Grep has no cache and reads no clock, so a call's result depends only on its arguments and the VFS contents. + - Read refuses a file over 16 MiB and never truncates it. Grep skips a file over 16 MiB from the size in the listing, and never fails on a single entry. + - A claim conflict arrives as `VfsError::Conflict` (`crates/runvil-internal/vfs/src/handle/claims.rs`) and is shown as the conflict sentence, never as a not-found message. + - File text is returned as untrusted output through `ToolOutput::untrusted`, and every access goes through `Access` so Mode and policy apply. + + + + +## Testing Plan + +Unit tests pin every text, boundary, and constant. Integration tests flip the stub expectations for Read, ListDir, and Grep to real output over `TestCall`'s in-memory store, and the yield check polls the Grep future by hand. No test in `plugin-std` needs the vfs crate. The exit criteria are the full gate set. + +- Unit: + - Read: the numbering window table; the cap and a page-by-page rebuild across 1, 2, 5, 37, 2,000, and 2,001 lines with and without a final newline; the first-line cut on lines of 1-, 2-, 3-, and 4-byte characters (99,992 or 99,990 kept bytes, always valid UTF-8); a cap sweep from 12 to 60 bytes over mixed-width text; a file that ends mid-sequence; the offset table on a 10-line file; line totals; carriage returns; the decode ladder; the binary sniff boundaries (NUL at index 8,191 versus 8,192, 5% versus 4.9%); every failure text byte for byte; the argument rules; a trust test; and the description pin, which asserts the text contains `100000` and `16777216` and not `cat -n`. + - ListDir: the POSIX tree example in the parent design report (`runvil-design/specs/cursor/agent-file-tools.md`), an empty directory, a file as target, a missing path, `.git` skipped while `.env` and `.gitignore` are listed, sort order against `Access::glob`'s for the same tree, the byte cap and its note, and `ignore_globs` matching at any depth. + - Grep: full-path order on `/B`, `/a-b`, `/a.txt`, `/a/x`, `/b`; `head_limit` global across files; page joins at 25 matches; `-C` overridden per side; merged context; the window around a match at position 1,001; skips with their counts; each regex failure; every argument string; UTF-8 safety of the long-line window and the context clip at every start offset over emoji and CJK text; and the description pin. + - Matcher: each grammar rule, dotfiles, any-depth matching, and each refusal. The cases the matcher shares with `Access::glob` are pinned against that function's own cases, and a test records the one difference, which is any-depth matching. + - Refusal text: every `VfsError` variant, including `Conflict`, built directly with no second call. + - Worked examples to reuse as cases are in `specs/cursor/master-read.md` and `specs/cursor/master-grep.md` in the runvil-design repository. Where one disagrees with a rule in this plan, this plan governs. + - VFS, in the vfs crate: `read_range` is denied by a read-denying policy, takes a read claim, fires the read event, and passes a `len` of 16,777,217 to a counting backend. +- Integration and end-to-end: + - Flip Read, ListDir, and Grep in `crates/plugin-std/tests/it/tools.rs` from the stub text to real output. The four remaining stubs keep their stub tests. + - Yield: seed 1,000 small files through the call context's `Access`, poll the Grep future by hand, assert it returns `Pending` at least 31 times before it completes, and assert that dropping it after the first `Pending` is safe. `TestCall` cannot supply a chosen `VfsRef` or an event sink (`crates/runvil-plugin/src/testing.rs`), so the test counts `Pending` polls and not reads. + - Real directory: one test over a directory mounted with `RealBackend::rooted`. It cannot live in `plugin-std`, so it goes in an existing test suite whose crate may depend on both `plugin-std` and the vfs crate. The executing agent picks that suite. +- Regression, security, and performance: + - The four stub tools still return the stub text. + - File text is untrusted output, and Grep's bare `No matches found` line is the one trusted exception. + - Cuts are checked over text of 1-, 2-, 3-, and 4-byte characters for Read's first line, Grep's long-line window, and Grep's context clip, so no cut splits a code point. + - Limits (files considered, bytes read, per-file size, slice size) are lowered through their parameters so tests reach each limit without large fixtures. +- Exit criteria: + - `cargo nextest run -p plugin-std` passes, then the full set as listed in `AGENTS.md`: `cargo nextest run --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-features` and `cargo nextest run --locked -p workshop -p workshop-server -p workshop-server-api`. + - Both clippy runs pass with `CARGO_BUILD_WARNINGS=deny`: `cargo clippy --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-targets --all-features` and `cargo clippy -p workshop -p workshop-server -p workshop-server-api --all-targets`. `cargo fmt --all --check` passes. + - The rustdoc run (`cargo doc --workspace --no-deps --all-features` with the workshop crates excluded and `RUSTDOCFLAGS="-D warnings"`) passes, and so does `cargo test -p build-xtask`. + - `node --test crates/workshop/ui/test/docs-claims.mjs` passes. + - `cargo + xtask api --check` passes with `CARGO_BUILD_WARNINGS` unset, where the nightly is the one named in `crates/build-xtask/src/api/toolchain.rs`, because the `Access` additions change the facade. + + + + +## Decision Record + +- Decisions: + - Name and ids. The Pack is `std` (crate `plugin-std`), Lua ids are `std/`, and model-facing names copy Cursor's. Rationale: one namespace for the file tools with the names models already know. The maintainer said: "what about std instead of core?", "std yes; the Lua names would be maybe std/read, std/write, std/grep, std/replace, std/rm ? the model facing names should match Cursor". + - Access only through the VFS, with no shell and no user approval yet. Rationale: Mode and policy then gate every access for free. The maintainer said: "VFS for sure; leave out shell, we can decide it later; no user approval, we will add that later". + - Text only. The maintainer said: "text only of course". + - Read and Grep behavior is the behavior in this plan, which follows the Read and Grep design reports in the runvil-design repository (`specs/cursor/master-read.md` and `specs/cursor/master-grep.md`) over Cursor's behavior. Rationale: the reports depart from Cursor on purpose (Read's cap is no longer an error, and Grep takes the bundle schema plus `literal`). The maintainer attached them as the context for this plan: "context: @runvil-design/specs/cursor/master-grep.md @runvil-design/specs/cursor/master-read.md". Where a report disagrees with this plan, this plan governs. + - No cut splits a UTF-8 code point. Rationale: a split sequence turns into replacement characters or an invalid string. The maintainer said: "Read has to make sure not to split a UTF code point. Just update the design files". + - Dotfiles are listed and searched, and only `.git` is skipped. Rationale: a model often needs to edit `.gitignore` and similar files. The maintainer said: "for the record, filtering dot files is stupid. one of the most useful things a model can do is stuff like edit .gitignore, and so on." + - ListDir is added as `std/dir` with the shape in the Functional Specification. Rationale: Cursor explores directories through its shell and keeps a directory tool only behind a deprecated setting, so a Runvil model with no shell needs one. The maintainer asked "what about std/dir" and said "cursor has ListDir, and I am working on adding it to the report". + - ListDir's wire name is `ListDir`, not Cursor's `LS`. Rationale: `LS` is deprecated and gated behind the setting `enableDeprecatedLsTool` (default false) in Cursor, so there is no live convention to match, and `ListDir` says what it does. This is the one exception to copying Cursor's wire names. The maintainer chose it when asked which name governs: "1. ListDir". + - `Access::canonical_path` is not added in this work. Rationale: nothing here calls it, because the read record that would use it ships with the real Write tool, and it would add public API with no caller. The maintainer said: "2. defer canonical_path". + - Grep is a pure `find` in the Pack over `Access::list` and `Access::read`, with no VFS search, no cache, no clock, no `` envelope, and no newest-first sort. Rationale: deterministic results, and a search that can be dropped between slices. + - Read uses a bounded `read_range` call after a `stat`. Rationale: the file's real size is never read when it is over the ceiling. + - The glob matcher is the Pack's own, with the any-depth rule for relative patterns and the same refusals as `Access::glob`. Rationale: Grep's `glob` and ListDir's `ignore_globs` need any-depth matching, and `Access::glob` anchors at the root. + - The Pack sorts each listing itself. Rationale: `Access::list` promises no order. + - Ignore rules ship as option B: skip any path with a segment named `.git` and apply no `.gitignore`. Rationale: the shared ignore module is separate work, and nothing real is mounted yet, so nothing is exposed. The maintainer said: "3. option B". + - Each tool's source lives in its own directory named after its Lua id, with the entry file beside the directory. Rationale: the repository retired `mod.rs` files in commit `128ff3a35`. The maintainer wrote a crate rule that each tool's source goes in its own directory, and then asked "is the three-file rule doing any actual work? ... maybe we should delete the three-file rule and just let everything accumuluate into the one dir?" and said "yes, make the replacement wording for AGENTS.md right now". + - Existing historical plans under `vibe/` are not edited. The maintainer said: "no, do not touch the vibe/ plan. that is historical data." + - Execution is four steps, one commit each: (1) `Access::read_range` with its VFS tests and regenerated `crates/runvil/public-api.txt`; (2) the shared `plugin-std` code and real Read; (3) the glob matcher and real ListDir; (4) real Grep with the full gate set. Rationale: few large steps keep the run short. The `read_range` commit stays separate because it regenerates the public API file, and the last step's full verification replaces a separate gates step. The maintainer said: "keep the number of steps low, dont bloat the plan with too many steps". + - Tests inside `plugin-std` use `TestCall`'s in-memory store. Rationale: a Plugin crate may not depend on the vfs crate, and `TestCall` cannot take a chosen `VfsRef`. +- Rejected alternatives: + - Cursor's Read cap as an error. Rejected because a cut read with a next-offset trailer lets the model continue. Revisit: never planned. + - A binary-extension list for Read. Rejected in favor of the content sniff. Revisit if the sniff misclassifies real files in a deployed run. + - The live Grep schema with `context` and `-n` and defaults `files_with_matches` and 250. Rejected for the bundle schema plus `literal`. + - A regex-translated glob matcher with braces and case folding. Rejected for the four-token grammar. Revisit if models need brace expansion. + - Grep through `Access::glob` plus whole-file `read_string`, or through a VFS search. Rejected for the pure `find`. Revisit when a deployed run shows searches over 2 seconds, which would justify a VFS search with a resumable step. + - `mod.rs` entry files. Rejected because the repository retired them. + - An entry file inside the tool directory with `#[path = "read/read.rs"] mod read;`. Rejected for the standard layout. Revisit if the maintainer prefers it. + - Cursor `LS` extras (extension-count summaries, the 5-second timeout, the terminal-log special case, the `Error listing directory:` prefix). Rejected for the shared cap and refusal text. + - Prepending `**/` to each `ignore_globs` pattern. Rejected because the matcher already matches relative patterns at any depth. + - `LS` as ListDir's wire name. Rejected for `ListDir`. Revisit if Cursor stops deprecating `LS`. + - Option A for ignore rules, which pulls the shared ignore module forward and first chooses between the `ignore` crate and a matcher of our own. Rejected for option B. Revisit when a real repository is mounted. + - A yield test and a two-call conflict test built over a chosen `VfsRef` with an event sink inside `plugin-std`. Rejected because the dependency boundary forbids the vfs crate there. Revisit if `TestCall` gains a way to share a `VfsRef`. +- Assumptions, risks, and notes: + - The numbers (100,000 and 15,000 bytes, 16 MiB, 100,000 files, 1 GiB, 32 files per slice, 2 MiB per slice, the 8,192-byte sniff, the 1,000-byte window) are judgments, not measurements. Each ships as one constant with a test. + - ListDir uses the shared 15,000-byte cap, Cursor's tree text, Runvil's sort, and the shared refusal wording instead of `LS`'s 2500 and 10,000 character budgets. No design report covers it. Confidence: medium. + - Grep's `type` table is about fifteen common names, not ripgrep's 224. Confidence: medium. + - A Grep future that returns `Pending` once reaches the Host's executor, by reading `crates/harness-internal/runner/src/host-run.rs` and `effect_loop.rs`. The re-poll behavior of the collection that holds the future was assumed from standard behavior, not read, and the hand-polling test exercises only the future. If a yield proves not to work, Grep needs the VFS search with a resumable step, and this plan changes. + - Without `.gitignore` pruning, a mounted real repository lets `target/` and `node_modules/` spend Grep's 100,000-file budget. The descriptions say `.gitignore` rules are not applied, and the zero-match line says no ignore rules ran. + - The Grep design report says "files considered" in its limit rule and "after reading" in its notice. This plan uses "considered" for both. + - A cross-scope claim conflict can be built in the vfs crate with two `acquire` calls on one `VfsRef` (`crates/runvil-internal/vfs/src/handle/tests/claims.rs`). A same-round sibling conflict needs an Engine-only function, so neither is reachable from `plugin-std` tests. + - The `Access::read_range` addition touches the public API file, which needs the pinned nightly to regenerate. + - The root `AGENTS.md` carries an edit that replaces the three-file layout rule with a looser one. The maintainer commits it before the run starts, because the run requires a clean worktree, and it belongs to no step. + - Existing `#[path = "..."]` attributes stay. A count today is about 252 in `src/` trees (185 point at test files), and cleaning them is not worth a dedicated change. Confidence: high - no behavior gain for about 200 touched files. + - Cursor's directory tool is named `LS` and is gated by `enableDeprecatedLsTool` (default false). This is from Cursor's own tool context and is not a repository fact. + +### Deferred and Out of Scope + +- Deferred: the read record (hash, shown range, total, whole flag), its per-run service, and its `needs` entry ship with the real Write tool. +- Deferred: the shared ignore module (the `ignore` crate versus a matcher of our own, which also decides whether `ignore` and `globset` join the lock file) and real Glob ship together. Revisit when a real repository is mounted, because until then `.gitignore` rules are not applied. +- Deferred: `Access::canonical_path`, which canonicalizes a path against the access root with no policy check, claim, or event. Revisit when the read record ships with the real Write tool. +- Deferred: a two-call conflict test (a Read against another call's write claim in one round). Revisit when Write is real or when `TestCall` can share a `VfsRef`. +- Deferred: a VFS search for Grep. Revisit when a deployed run shows searches over 2 seconds. +- Deferred: real Write, StrReplace, Delete, and Glob, and user approval. +- Deferred: the separate VFS todos `vfs-glob-gaps`, `vfs-mode-text`, `vfs-write-mode`, and `vfs-plan-subtree`, and the Engine todo `engine-cut-guard`, all listed in `vibe/2026-10/2026-10-27-2-plugin-std-file-tools.md`. +- Deferred: mounting a real directory in Workshop, which is a product decision about which directory to mount. +- Out of scope: a shell Pack, PDFs, images, and any change to the Engine or the Harness. +- Out of scope: any edit to an existing file under `vibe/`. + + + + +## Project Survey + +- Status: complete +- Build command: `cargo build --locked -p `; a plain `cargo build` builds only the default members (`runvil`, `runvil-plugin`, `harness`, `harness-gateway-client`). The desktop app is opt-in with `cargo build -p workshop`. Cargo aliases in `.cargo/config.toml`: `cargo workshop`, `cargo xtask`, `cargo vibelight`. The pinned toolchain is Rust 1.99 (`rust-toolchain.toml`), edition 2024, MSRV 1.95. +- Focused test command pattern: `cargo nextest run --locked -p ` (cargo-nextest 0.9.128 is installed; nextest profiles live in `.config/nextest.toml`, with a `ci` profile). +- Component test command pattern: `cargo nextest run --locked -p ` for a whole crate, or `cargo nextest run --locked -p --test it` for a crate's integration target (`it` for plugin crates, `suite` for `harness` and `runvil`). For `plugin-std` that is `cargo nextest run --locked -p plugin-std`. +- Full-suite test command: `cargo nextest run --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-features`; the Workshop crates run separately with `cargo nextest run --locked -p workshop -p workshop-server -p workshop-server-api`. Structural checks: `cargo test -p build-xtask`; wording scan: `node --test crates/workshop/ui/test/docs-claims.mjs`; facade surface: `cargo + xtask api --check` (nightly named in `crates/build-xtask/src/api/toolchain.rs`; run it with `CARGO_BUILD_WARNINGS` unset). +- Linter command: `cargo clippy --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-targets --all-features`, run with `CARGO_BUILD_WARNINGS=deny`; Workshop crates: `cargo clippy -p workshop -p workshop-server -p workshop-server-api --all-targets`. Clippy covers `cargo check`, so never run a standalone `cargo check --workspace`. The workspace lints (`[workspace.lints]` in the root `Cargo.toml`) deny `unwrap_used`, `expect_used`, `todo`, `allow_attributes`, and `unsafe_code`, and warn on `missing_docs`; tests may unwrap and expect (`clippy.toml`). +- Formatter check command: `cargo fmt --all --check` (`rustfmt.toml` sets `style_edition = "2024"`; the pre-commit hook runs this). +- Docs command: `RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --all-features --exclude workshop --exclude workshop-server --exclude workshop-server-api`; facade docs: `RUSTDOCFLAGS="-D warnings" cargo doc -p runvil --no-deps` (CI also runs `cargo doc -p harness --no-deps`). Rustdoc lints are not covered by clippy. +- Test placement and naming conventions: + - Unit tests sit in a sibling file named `-tests.rs`, wired with `#[cfg(test)] #[path = "-tests.rs"] mod tests;` (`plugin-std` has `check-tests.rs`, `pack-tests.rs`, `spec-tests.rs`). New code adds no `#[path]` attribute: a module with more than one file lives in a directory named after it (`foo.rs` beside `foo/bar.rs`) and its unit tests are `foo/tests.rs`. Larger groups use a `tests/` directory under `src` (for example `runvil-vfs/src/handle/tests/*.rs`), and shared helpers sit in `test_support.rs`. + - Integration tests are a single Cargo target in a directory: `tests/it/main.rs` plus topic modules (`plugin-std/tests/it/{main,tools}.rs`), or `tests/suite/main.rs` for `harness` and `runvil`. + - A test file that panics on setup uses `#![expect(clippy::expect_used, reason = "...")]`; every `allow` or `expect` attribute needs a `reason`. + - Every `src`, `tests`, `benches`, and `examples` Rust file must stay at or under 500 lines (each crate's `build.rs` runs the `build-ceiling` check, which fails the build otherwise). + - Engine crates (`runvil` and `runvil-*`) never sleep, read the wall clock, wait with a timeout, or draw OS entropy in tests, use only seeded pseudo-random input, and name temp directories from the test name, process id, and a counter. `plugin-std` is not an Engine crate but reaches files only through the call's `Access`. +- Directory map: + - `crates/`: the workspace. Top level is the public and shared layer (`runvil`, `runvil-plugin`, `harness`, `harness-gateway-client`, `plugin-mcp`, `plugin-std`, `plugin-user-input`, `plugin-web`, `shared-error-source`, and `build-*` tooling crates). Private families live in manifestless containers: `runvil-internal/` (Engine: `engine`, `lua`, `model-client`, `parser`, `types`, `vfs`), `harness-internal/` (`runner`, `plugin-loader`, `test-plugin`), and `workshop/` (Workshop crates plus `ui`). + - `.cargo/`, `.config/`, `.githooks/`: Cargo aliases and Windows linker flags, nextest profiles, and the pre-commit (fmt) and pre-push (docs-claims, clippy, `cargo deny`) hooks. + - `.github/`: workflows (`ci.yml`, `nightly.yml`, `release-workshop.yml`, `site.yml`, `workshop-installer-smoke.yml`) and fixtures. + - `.cursor/rules/`: Workshop architecture and SPA rules. `.vibelight/`: generated per-crate outline files (md and html), configured by `vibelight.toml`. + - `guide/`: mdBook sources (`books/language`, `books/workshop`) and landing page. `prompts/`: example prompt files. `images/`: banner images. + - `tools/`: Node `.mjs` scripts (with `.test.mjs` files) that pin, fetch, and stage the Gateway sidecar. `gateway-pin/`: `pin.json` for the Gateway release Workshop uses. + - `vibe/`: dated plan and work notes, grouped by month. `local/`: untracked scratch and logs (gitignored). `target/`: build output (gitignored). + - Root files: `AGENTS.md` (definitions, principles, verification gates), `Cargo.toml`, `Cargo.lock`, `clippy.toml`, `deny.toml`, `rustfmt.toml`, `rust-toolchain.toml`, `README.md`, `LICENSE` (BSL-1.0). +- Component boundaries: + - Dependency directions (from each crate's `[dependencies]`): `runvil-types` and `runvil-vfs` depend on nothing in the workspace; `runvil-model-client` depends on `runvil-types`; `runvil-lua` on `runvil-model-client`, `runvil-types`, `runvil-vfs`; `runvil-parser` on `runvil-lua`, `runvil-types`; `runvil-engine` on all of those; the `runvil` facade on the Engine crates; `runvil-plugin` on `runvil-types` and `runvil-vfs`. + - Plugin crates (`plugin-std`, `plugin-mcp`, `plugin-user-input`, `plugin-web`) depend only on `runvil-plugin` (plus `shared-*` and outside libraries); they may name no `build-*` crate, which is why `plugin-std/build.rs` includes the `build-ceiling` source by path. + - `harness-runner` and `harness-plugin-loader` sit under `harness`; `harness` depends on `runvil` and `runvil-plugin`. Workshop crates depend on the Harness, the plugin crates, and `runvil`; code outside the Engine reaches `runvil-internal` only through `runvil` and `runvil-plugin`. `cargo test -p build-xtask` enforces these boundaries, the Workshop tier graph, the `## Invariants` marker, and lint inheritance. +- Conventions summary: + - Every crate inherits `version`, `edition`, `license`, and `repository` from `[workspace.package]`, takes dependencies with `.workspace = true`, sets `[lints] workspace = true`, and carries a `build.rs` (file-size check). Comments in `Cargo.toml` explain each pin. + - Crate-level docs live in `lib.rs` (`//!`) and end with a `## Invariants` list; docs name `Engine`, `Harness`, `Host`, `Pack`, `Plugin`, `Static Set`, and `Dynamic Set` with one meaning each, and Engine crate docs say "the caller", never the Host. The wording rules in `AGENTS.md` are enforced by `docs-claims.mjs`. + - A tool's source goes in its own directory named after its Lua name (`plugin-std/AGENTS.md`); tool definitions are JSON under `schemas/`, embedded at build time, with edits recorded in `schemas/EDITS.md`. + - Behavior changes ship with tests in the same change; error messages are written for a model to read (concise, factual, required versus actual); no unsafe code outside an owned boundary; process-global installers are banned outside binary entry points (`clippy.toml`). + - JSON that reaches a recorder round-trips exactly (sorted keys, finite numbers, `float_roundtrip`). Cargo features gate only real constraints. A change to the facade's public surface regenerates `crates/runvil/public-api.txt`. + - Code must build and test on Windows and Unix: CI runs Windows and Linux jobs, and `AGENTS.md` gives the PowerShell form for unsetting `CARGO_BUILD_WARNINGS`. + + + + +## Execution Instructions + + + +### Step 1: Add Access::read_range to the VFS [completed] + +- Component: `vfs-read-range` +- Tests: + - `cargo nextest run --locked -p runvil-vfs` + - `cargo fmt --all --check` + - `RUSTDOCFLAGS="-D warnings" cargo doc -p runvil --no-deps` + - `cargo +nightly-2026-09-05 xtask api --check` + +Run the api check with `CARGO_BUILD_WARNINGS` unset (PowerShell: `Remove-Item Env:CARGO_BUILD_WARNINGS`). The nightly name must match `PINNED` in `crates/build-xtask/src/api/toolchain.rs`; if that file names another nightly, use it. + +Placement: this is the first component because Read in step 2 calls `Access::read_range` and nothing else in this plan changes the VFS. It is one piece built in one step: the method, its four tests, and the regenerated API listing are one behavior. It is its own commit because it changes `crates/runvil/public-api.txt`, which needs the pinned nightly. + +Build: + +- `crates/runvil-internal/vfs/src/handle/operations.rs`: add `pub fn read_range(&self, path: &str, offset: u64, len: u64) -> Result, VfsError>` on `Access`, right after `read_string`. It takes the same four steps as `read`, in this order: `let path = self.gate(Op::Read, path)?;`, `self.admit(Claims::claim_read, &path)?;`, `self.fire(Op::Read, &path);`, then `self.inner().read_range(&path, offset, len)`. The doc comment says it returns up to `len` bytes from byte `offset`, fewer at the end of the file, and nothing at or past the end. It says `RealBackend` seeks and reads only the requested bytes, and that a backend without an override reads the whole file and slices it, so it is not bounded there. Its `# Errors` section matches `read`. The doc text must not name an internal crate, because `cargo xtask api --check` refuses that. +- Nothing else in the VFS changes. `RealBackend::read_range` (`real.rs`), the forwarding impls (`handle/forward.rs`), and `router-access.rs` already implement or forward `VfsAccess::read_range`. Keep `operations.rs` at or under 500 lines. +- `crates/runvil/public-api.txt`: regenerate with `cargo +nightly-2026-09-05 xtask api --bless` (`CARGO_BUILD_WARNINGS` unset). The diff must be exactly one new line, between the `Access::read` and `Access::read_string` lines: `pub fn runvil::vfs::Access::read_range(&self, path: &str, offset: u64, len: u64) -> core::result::Result, runvil::vfs::VfsError>`. + +VFS tests, in a new file `crates/runvil-internal/vfs/src/handle/tests/read_range.rs` declared as a module in `handle/tests.rs` (no `#[path]` attribute). Model each on the named existing test: + +- Denied by policy: with a policy that denies `Op::Read` and gives a reason that names the path, `read_range("/f.txt", 0, 10)` returns `VfsError::PermissionDenied` with that reason, and the backend call log (`TestFs` in `test_support.rs`) holds no read. Model: `a_policy_denied_operation_never_fires_the_sink` in `handle/tests/sink.rs`. +- Takes a read claim: after one identity calls `read_range("/f.txt", 0, 4)`, a write to `/f.txt` by another identity in another scope fails with `VfsError::Conflict`, and two identities that both call `read_range` on one path never conflict. Model: `a_write_conflicts_with_another_identitys_read_claim` and `reads_by_two_identities_never_conflict` in `handle/tests/claims.rs`. +- Fires the read event: a handle with an `on_op` sink receives one `OpEvent` with `Op::Read` and the canonical path for a `read_range`, and a denied `read_range` fires none. Model: `a_sink_receives_events_in_order_with_op_path_and_label`. +- Passes the length through: a small test backend, defined in the test file, whose `VfsAccess::read_range` override records its `offset` and `len`. `read_range("/f.txt", 3, 16_777_217)` makes it record `(3, 16_777_217)`, so the call is neither clamped nor turned into a whole-file read. Do not change `TestFs`, whose session deliberately implements only the required methods. +- Slicing on `MemoryBackend`, through the handle: on `hello world`, `(6, 5)` returns `world`, `(6, 100)` returns `world`, and `(100, 5)` returns nothing. + +Do not touch the Engine, the Harness, `plugin-std`, or any file under `vibe/`. + +Commit: one commit holding the method, its tests, and `crates/runvil/public-api.txt`. Suggested subject: `Add Access::read_range to the VFS`. + + + + + +### Step 2: Shared plugin-std code and real Read [completed] + +- Component: `read-tool` +- Tests: + - `cargo nextest run --locked -p plugin-std` + - `cargo clippy --locked -p plugin-std --all-targets -- -D warnings` + - `cargo fmt --all --check` + - `node --test crates/workshop/ui/test/docs-claims.mjs` + +Placement: second, because Read calls `Access::read_range` from step 1, and because ListDir and Grep reuse the shared code this step adds (refusal text, argument helpers, `plural`, the text module). The component has two pieces, the shared foundation and the Read tool, built jointly in one step: the foundation has no caller but Read until step 3, and Read's failure texts and decoding are only testable through the foundation. + +Rules for every step in this plan: warnings are denied and clippy runs `pedantic`, so add no item before something uses it (an unused `pub(crate)` item fails the build) and give every item a doc comment. Tests may use `unwrap` and `expect`. New code adds no `#[path]` attribute: a module is `foo.rs` beside a `foo/` directory, with its unit tests in `foo/tests.rs` (split into a `tests/` directory by topic when a file nears 500 lines). Every Rust file stays at or under 500 lines. Do not touch the Engine, the Harness, the vfs crate, or any file under `vibe/`. Do not make `plugin-std` depend on the vfs crate: import `Access`, `VfsError`, `Entry`, `FileType`, and `Stat` from `runvil_plugin`. + +Dependency and lock file: + +- `crates/plugin-std/Cargo.toml`: add `regex.workspace = true` with a one-line comment (Grep compiles its pattern with Rust regex syntax). The root `Cargo.toml` already holds `regex = "1"`, so it does not change. Build once without `--locked` so `Cargo.lock` records the new edge, and commit `Cargo.lock`. Nothing uses `regex` until step 4, and no unused-dependency check exists. + +Shared code, all in `crates/plugin-std/src/`: + +- `refusal.rs` (unit tests in `refusal/tests.rs`). `pub(crate) fn refusal(tool: &str, path: &str, error: &VfsError) -> String`, where `tool` is the wire name and `path` is the path as the model wrote it, never the canonical path. It returns one model-readable sentence per `VfsError`: + - `NotFound` gives `File not found: {path}`. + - `IsADirectory` gives `Path is a directory, not a file: {path}`. + - `NotADirectory` gives `Path is not a directory: {path}`. + - `Conflict` gives `{path} was changed, or is being changed, by another call in this round, so it cannot be read in this round. Call {tool} again in your next message.` Nothing retries inside a call. + - `Io { kind, message, .. }` gives `I/O failure on {path} ({kind}): {message}`, with `kind` printed through `Display`. + - `PermissionDenied { reason, .. }` and `ApprovalRequired { reason, .. }` give the reason unchanged. + - Every other variant, and any variant added later (the enum is `#[non_exhaustive]`, so keep a wildcard arm), gives the variant's `Display` text with its leading label dropped: the text after the first `": "`, or the whole text when it has none. For example `OutsideRoot` reads as its `detail`. + - A message names no other tool. + - Also add `pub(crate) fn neither_file_nor_directory(path: &str) -> String` giving `Path is neither a file nor a directory: {path}`, and `pub(crate) fn missing_root(path: &str) -> String` giving `Path does not exist: {path}` (step 3 and step 4 use it for a missing search root; add `missing_root` only when step 3 first calls it, to avoid dead code). + - Tests: build every `VfsError` variant directly as a struct literal, with no second call, and assert the sentence byte for byte, including `Conflict`. Assert that a relative path `notes/a.md` is echoed as written and not as `/notes/a.md`, that a denied read passes a reason that names the path through unchanged, and that no message names `Write`, `StrReplace`, `Delete`, `Glob`, `Grep`, or `Shell`. +- `check.rs` (keep the existing `check` and its `check-tests.rs` sibling; add tests there). Add three helpers: + - `refuse_unknown(tool, schema, args)` refuses a key the schema's `properties` does not declare, in the shape ``std_{Wire}: `{key}` is not a parameter of this tool. The parameters are `a`, `b` and `c`.`` The names are sorted bytewise (serde_json keeps no insertion order here, so sort explicitly rather than rely on feature unification), joined with `, ` and with ` and ` before the last, and a single name has no `and`. The error kind is `InvalidArguments`. It runs after `check` and before any tool logic. The four stub tools do not call it. + - A blank-path helper: ``std_{Wire}: `{name}` is empty or only whitespace, and the call passed "{value}"`` (kind `InvalidArguments`), returned when the value is empty or whitespace-only. It runs before any I/O. + - `plural(n: usize, noun: &str) -> String` returns `{n} {noun}` for any `n` but 1, with `s` added, and `1 {noun}` for 1. A noun ending in a consonant plus `y` takes `ies` (`directory` gives `directories`, and `entry` gives `entries`). Examples: `1 line`, `10 lines`, `0 lines`. +- `text.rs` (unit tests in `text/tests.rs`). Public to the crate: `decode`, `sniff_binary`, `split_lines`, `clip_to_boundary`, and the constant `MAX_FILE_BYTES: u64 = 16_777_216`. Grep calls the same code in step 4, so a Grep line number names the line Read prints. + - `decode(bytes: &[u8])` drops a leading UTF-8 mark (`EF BB BF`). A file that starts with `FF FE` or `FE FF` is decoded with `String::from_utf16_lossy` after dropping the mark (little-endian for `FF FE`, big-endian for `FE FF`), and the result records `UTF-16LE` or `UTF-16BE`. Anything else is decoded as UTF-8 lossily, and the result counts the invalid sequences (one per invalid chunk from `<[u8]>::utf8_chunks`, so a file that ends mid-sequence counts one). It returns the text, the UTF-16 name if any, and the invalid count. Never use `read_string` and never return `VfsError::NotUtf8` to the model. Do not guess any other encoding. + - `sniff_binary(bytes: &[u8]) -> bool` looks only at the first 8,192 bytes. It is true when any of them is NUL, or when at least 5% of them are control bytes (`u8::is_ascii_control`, which is below 0x20 and 0x7f) other than tab, LF, and CR; compare with integers, `controls * 20 >= sampled_len`. A file that starts with a UTF-16 mark skips the sniff, because UTF-16 text holds NUL bytes by design; UTF-16 without a mark is binary. Boundary tests: a NUL at index 8,191 is binary and at index 8,192 is not; in a 1,000-byte sample 50 control bytes (5%) is binary and 49 (4.9%) is not; tab, LF, and CR never count. + - `split_lines(text: &str) -> Vec<&str>` splits on `\n`, drops the empty element after a final newline, and strips one trailing `\r` from each line. So `a\nb\n` and `a\nb` both give two lines, an empty text gives none, and `a\r\nb\r\n` gives `a` and `b`; a lone `\r` inside a line is kept, so `first\rNEEDLE\rthird` is one line. + - `clip_to_boundary(text: &str, start: usize, end: usize) -> &str` is the only function that shortens text inside a line. It rounds `start` up and `end` down to the nearest `str::is_char_boundary` position, clamps both to the length, and returns the sub-slice. It never slices by a raw byte index without that check. A prefix cut is `clip_to_boundary(text, 0, max)`. + +Read, in `crates/plugin-std/src/read.rs` (entry) and `src/read/` (`args.rs`, `window.rs`, `render.rs`, `tests.rs` and a `tests/` directory as needed): + +- Entry: `pub(crate) fn call(access: &Access, definition: &Definition, args: &Value) -> Result`. Read does no yielding. Constant `pub(crate) const BYTE_CAP: usize = 100_000` in `read.rs`. The caps are parameters of `window` and `render` with the constants as defaults, so tests can lower them. +- Arguments: `path` (string, required), `offset` (integer), `limit` (integer). A `null` reads as absent and numeric strings are not coerced (the shared `check` already does both). Order: `check::check`, then `refuse_unknown`, then the blank-path helper, then `limit`, then I/O. A JSON integer that does not fit in `i64` is refused as ``std_Read: `offset` must be a JSON integer that fits in 64 bits, and the call passed {value}`` (same shape for `limit`). A `limit` below 1 is refused with ``std_Read: `limit` must be at least 1, and the call passed 0`` (the passed value is printed as given). +- Read order: (1) `access.stat(path)`; a stat failure becomes `refusal(...)`. A `Directory` gives `Path is a directory, not a file: {path}`. A `Fifo`, `Socket`, `CharDevice`, or `BlockDevice` gives `neither_file_nor_directory`. A `File` or `Symlink` goes on (the VFS reads through a link, and refuses it with `OutsideRoot` when it leaves the run's filesystem). Put the file-type decision in a pure function over `FileType` so a test reaches every variant. (2) `access.read_range(path, 0, 16_777_217)`; a failure becomes `refusal(...)`. If more than 16,777,216 bytes come back, fail with `File is too large to read: {path} is over the limit of 16777216 bytes.` Read never truncates a file over the ceiling. (3) `sniff_binary`; if true, fail with `File content is binary data, not text: {path}`. There is no extension list. (4) `decode`, then `split_lines`. +- Failures are `ToolError` values: argument errors with kind `InvalidArguments`, every other failure above with `ToolError::message(...)`. An empty result is a success: when there are no lines (a zero-byte file, or a file of only a byte order mark) the answer is `File is empty.` for any `offset` or `limit`. Every successful Read, including `File is empty.`, returns `ToolOutput::untrusted`. +- Window, in `window.rs`, over `N` lines with `N` at least 1: the default offset is 1, and offset `0` reads as 1. A positive offset above `N` fails with `Offset {offset} is beyond the end of {path}, which has {plural(N, "line")}.` (for example `Offset 11 is beyond the end of /notes.md, which has 10 lines.` and `Offset 2 is beyond the end of /one.md, which has 1 line.`). A negative offset starts at `max(1, N + offset + 1)` and never fails, so `-1` is the last line and with no limit it reads the last `|offset|` lines. The window ends at `N` with no limit, and at `min(N, start + limit - 1)` with one, so an overlapping window is clamped. On a 10-line file: no offset gives 1-10, `0` gives 1-10, `10` gives 10-10, `-2` gives 9-10, `-100` gives 1-10, `-2` with `limit` 5 gives 9-10, `limit` 3 gives 1-3, offset 4 with `limit` 3 gives 4-6, offset 9 with `limit` 5 gives 9-10, and `11` fails. +- Render, in `render.rs`. Each window line becomes `{number:>6}|{text}` plus `\n`: a six-wide right-aligned number, a pipe, no space, always on, widening past six digits. The cap is 100,000 bytes of those numbered lines, counting each prefix, the text, and its newline, and not the trailer or notes. Stop before the line that would pass the cap. After the numbered lines come the notes, then the trailer, each on its own line, joined by `\n` with no newline after the last one. The trailer is the last line of a cut read: + - Cut by the cap: `[Showing lines {first}-{last} of {total} (100000-byte limit). Use offset={last + 1} to continue.]`, for example `[Showing lines 1-2083 of 5000 (100000-byte limit). Use offset=2084 to continue.]`. + - Ended by the caller's `limit` before the last line of the file: the same text without the parenthesis, `[Showing lines {first}-{last} of {total}. Use offset={last + 1} to continue.]`. + - A read that reaches the last line of the file prints no trailer, which includes the last page and every negative-offset tail read. + - First-line cut: when the window's first line alone passes the cap, keep `clip_to_boundary(text, 0, cap - prefix_len - 1)` (99,992 bytes for a six-wide prefix and the default cap), print it numbered, show no other line, and make the trailer `[Line {n} is {bytes} bytes and is cut after {kept} bytes. No parameter reads the rest of a line.` followed by ` Use offset={n + 1} to continue.` when a later line exists in the file, then `]`. `bytes` is the line's full byte length and `kept` is the bytes kept, so ASCII gives `cut after 99992 bytes` and 3-byte characters give `cut after 99990 bytes`. This is the only cut inside a line. UTF-16 is decoded in full before the cut. + - Notes, in this order, before the trailer: a UTF-16 file adds `[Note: the file is UTF-16LE and was decoded to text.]` (or `UTF-16BE`), and replaced bytes add `[Note: {n} invalid UTF-8 sequences in this file were shown as U+FFFD.]` using `plural(n, "invalid UTF-8 sequence")` for the count and noun, so one reads `1 invalid UTF-8 sequence`. +- Worked checks to reuse: 5,000 lines of 40 `x` give 2,083 numbered lines, the cap trailer above, and a length of at most 100,120 bytes. More worked examples are in `runvil-design/specs/cursor/master-read.md`; where one disagrees with this step, this step governs. + +Definition and wiring: + +- `schemas/read.json`: `offset` and `limit` become `"type": "integer"`. Rewrite the description: remove the `cat -n` line and the 2000-character-line line, and state the six-wide `{number}|{text}` format and that the prefix is not part of the line, the cap of `{byte_cap}` bytes of numbered lines and the trailer that names the next offset, the `{file_ceiling}`-byte ceiling, that binary files are refused, that Read opens any path the call names inside what the run has mounted, and that a link is read through unless it leaves the run's filesystem. Keep Cursor's batching line and the `File is empty.` line. Update the `offset` text: 1 is the first line, 0 reads as 1, negative values count from the end, and `-1` is the last line. The description may name other tools but must not name PDFs, `Shell`, or the Cursor workspace (an existing test refuses those). +- The description is built from the constants the code uses. Keep the placeholders `{byte_cap}` and `{file_ceiling}` in `read.json`, and have `spec.rs` fill them when it builds each `Definition`, from `crate::read::BYTE_CAP` and `crate::text::MAX_FILE_BYTES` (a small function keyed by the tool's segment, so steps 3 and 4 add their own placeholders). The built description holds `100000` and `16777216`. +- `schemas/EDITS.md`: replace the `read.json` section so it records the new description, the removed `cat -n` and 2000-character lines, and that `offset` and `limit` are integers. +- `src/pack.rs`: in `Std::call`, after `crate::check::check(...)`, dispatch on `definition.segment`: `"read"` calls `crate::read::call(cx.access(), definition, &args)`, and every other segment keeps the stub. Add `mod read;`, `mod refusal;`, and `mod text;` in `lib.rs`. Update the crate docs in `lib.rs` and the doc comments in `pack.rs` that say every tool is a stub: Read is real, and Write, StrReplace, Delete, and Glob are still stubs that answer `NOT_IMPLEMENTED`. Keep the wording rules `docs-claims.mjs` enforces. + +Tests: + +- Unit, `text/tests.rs`: the decode ladder (plain UTF-8; a UTF-8 mark hidden; UTF-16LE and UTF-16BE with a mark decoded and named; UTF-16 without a mark is binary; `61 FF 62 0A` gives `a`, U+FFFD, `b` and one invalid sequence; a file that ends mid-sequence counts one), the sniff boundaries above, `split_lines` (`a\nb`, `a\nb\n`, empty, `a\r\nb\r\n`, `first\rNEEDLE\rthird`), and `clip_to_boundary` over 1-, 2-, 3-, and 4-byte characters at every start and end offset, always valid UTF-8 and always inside the range asked for. +- Unit, `refusal/tests.rs` and `check-tests.rs`: as listed under the shared code, plus `refuse_unknown` (names the key, lists the sorted parameters with ` and ` before the last), the blank-path text with `" "` as the value, and `plural` (`1 line`, `10 lines`, `0 lines`, `1 directory`, `2 directories`). +- Unit, `read/tests.rs` and its topic files (seed files through `TestCall::new(id).context().access()` with `Access::write`): + - The numbering window table above, line totals, and carriage returns. + - The cap and a page-by-page rebuild across files of 1, 2, 5, 37, 2,000, and 2,001 lines, each with and without a final newline: follow `Use offset=N` under caps of 100,000, 300, and 90 bytes and limits of none, 50, and 7, then assert the rebuilt lines equal the original and no page is empty. + - The first-line cut on a 150,000-byte ASCII line (99,992 bytes kept), 50,000 CJK characters (99,990 kept, `cut after 99990 bytes`), a line of 2-byte characters (99,992 kept), and a line of 4-byte emoji (99,992 kept, 24,998 characters), always valid UTF-8. + - A cap sweep from 12 to 60 bytes over a first line that mixes 1-, 2-, 3-, and 4-byte characters: at every cap the kept bytes are valid UTF-8, a prefix of the line, within the allowance, and the notice's count equals the kept bytes. + - A file that ends mid-sequence, the offset table on a 10-line file, a binary file (`File content is binary data, not text: {path}`), the file-type function over every `FileType`, and the empty-file answer with `offset: 500`. + - Every failure text byte for byte: not found for a missing path (also a relative `notes/a.md` echoed as written), a directory, too large (a file of 16,777,217 bytes), offset past the end, and the blank path. + - The argument rules: `limit` 0 and `limit` -3, a string `offset`, an undeclared key, and `null` read as absent. + - A trust test: every successful read is `OutputTrust::Untrusted`. + - The description pin, in `spec-tests.rs`: the built `read` description contains `100000` and `16777216`, does not contain `cat -n` or `2000 characters`, and holds no unfilled `{byte_cap}` or `{file_ceiling}`. +- Integration, `tests/it/tools.rs`: add a helper that seeds files through `call.context().access()` before the call. Remove `("read", "Read")` from `every_tool_answers_the_not_implemented_text_naming_itself` (the other five stay). Add Read tests: a seeded `/notes.md` of `hello\nworld\n` answers ` 1|hello\n 2|world\n` as untrusted output, a zero-byte file answers `File is empty.`, and a missing path fails with `File not found: /nope.md`. +- Not testable here and not required: a two-call claim conflict (the conflict text is covered by the refusal test). Do not add a test that needs the vfs crate. + +Commit: one commit holding the shared code, the Read tool, `read.json`, `EDITS.md`, `Cargo.toml`, `Cargo.lock`, the doc updates, and every test. Suggested subject: `Add shared std file-tool code and a real Read`. + + + + + +### Step 3: Glob matcher and real ListDir [completed] + +- Component: `dir-tool` +- Tests: + - `cargo nextest run --locked -p plugin-std` + - `cargo clippy --locked -p plugin-std --all-targets -- -D warnings` + - `cargo fmt --all --check` + - `node --test crates/workshop/ui/test/docs-claims.mjs` + +Placement: third. ListDir needs the shared refusal text, argument helpers, and `plural` from step 2, and Grep in step 4 needs the matcher, so the matcher lands here and step 4 starts with a tested matcher. ListDir is smaller than Grep and is the matcher's first caller, which keeps the matcher free of dead code. The component has two pieces, the matcher and the ListDir tool, built jointly in one step: the matcher's only caller this step is ListDir's `ignore_globs`, and its behavior is only visible through that call. + +Rules for every step in this plan: warnings are denied and clippy runs `pedantic`, so add no item before something uses it (an unused `pub(crate)` item fails the build) and give every item a doc comment. Tests may use `unwrap` and `expect`. New code adds no `#[path]` attribute: a module is `foo.rs` beside a `foo/` directory, with its unit tests in `foo/tests.rs` (split into a `tests/` directory by topic when a file nears 500 lines). Every Rust file stays at or under 500 lines. Do not touch the Engine, the Harness, the vfs crate, or any file under `vibe/`. Do not make `plugin-std` depend on the vfs crate: import `Access`, `VfsError`, `Entry`, `FileType`, and `Stat` from `runvil_plugin`. The ListDir tool is the seventh tool, appended after `grep`. + +Code from step 2 that this step uses, already in the tree: `src/refusal.rs` (`refusal(tool, path, &VfsError) -> String`, `neither_file_nor_directory`), `src/check.rs` (`check`, `refuse_unknown`, the blank-path helper, `plural(n, noun)`), and `src/text.rs` (`decode`, `sniff_binary`, `split_lines`, `clip_to_boundary(text, start, end)`, `MAX_FILE_BYTES`). `spec.rs` has a small fill function that replaces `{name}` placeholders in a tool's description with numbers taken from the code's constants. + +Matcher, `crates/plugin-std/src/matcher.rs` with tests in `matcher/tests.rs`. The crate cannot import the vfs matcher, so port its four-token grammar from `crates/runvil-internal/vfs/src/glob.rs` (read it first) and keep the same refusals: + +- API: `pub(crate) struct Matcher`, `Matcher::new(pattern: &str) -> Result`, and `Matcher::is_match(&self, path: &str) -> bool`. `path` is an absolute virtual path with a leading `/`. `PatternError` has a `reason()` text for each refusal. +- Grammar: only `*` and `**` are special. `*` matches zero or more bytes except `/` inside one segment and does match dotfiles. `**` as a whole segment matches zero or more whole segments. Every other byte is literal, including `?`, `[`, `{`, `!`, and `.`, and matching is case-sensitive. A pattern with no leading `/` matches at any depth (match it as if `**/` came first); a pattern with a leading `/` is a full path from the root. This differs from `Access::glob`, where a relative pattern is anchored at the access root. +- Refusals, with these reason texts: empty (`pattern is empty`), longer than 1,024 bytes (`pattern is longer than 1024 bytes`), a control byte below 0x20 or 0x7f (`pattern contains a control character`), a backslash (`pattern does not support backslash escapes`), more than two `*` in a row (`more than two consecutive '*' are not supported`), and `**` as less than a whole segment (`'**' must occupy a whole path segment`). The argument error is ``std_{Wire}: `{name}` is invalid: "{pattern}" does not parse: {reason}.`` with `{name}` the parameter (`ignore_globs` here), and the echoed pattern cut to 80 bytes with `clip_to_boundary` and `...` added when it was cut. Build this text in `check.rs` beside the other argument helpers. +- Tests: each grammar rule, dotfiles matched by `*`, any-depth matching, and each refusal. Pin the cases it shares with `Access::glob` against that function's own cases from `glob.rs`, run anchored with a leading `/` on the pattern and on each path: the validation table (1,025 `a` is too long; `a\u{0}b` is a control byte; `a\b` is a backslash; `a/***/b` and `a/**b` are wildcard refusals; `ok/*.txt` is fine) and the match table over the paths `a.txt`, `src/a.rs`, `src/b.rs`, `src/deep/c.rs`, `src/deep/deeper/d.rs`, `notes/today.md`: `*.txt` gives true, false, false, false, false, false; `src/*.rs` gives false, true, true, false, false, false; `src/**/*.rs` gives false, true, true, true, true, false; `**/*.md` gives false, false, false, false, false, true; `src/**` gives false, true, true, true, true, false; `no*match` gives all false. Add one test that records the single difference: unanchored `*.txt` also matches `/x/y/a.txt`, which `Access::glob("*.txt")` would not. + +ListDir, in `crates/plugin-std/src/dir.rs` (entry) and `src/dir/` (`args.rs`, `walk.rs`, `render.rs`, `tests.rs` and a `tests/` directory as needed): + +- Entry: `pub(crate) fn call(access: &Access, definition: &Definition, args: &Value) -> Result`. No yielding: the byte cap bounds the work. Add `pub(crate) const BODY_CAP: usize = 15_000` to `text.rs`, the cap Grep shares in step 4. The cap is a parameter of the walk and render functions with this constant as the default. The walk takes its directory lister as a parameter (a closure over `Access::list`), so a test can supply a `Symlink` entry that the in-memory backend cannot make. +- Arguments: `target_directory` (string, required) and `ignore_globs` (array of strings). Order: `check::check`, `refuse_unknown`, the blank-path helper (a blank `target_directory` is refused before any I/O: ``std_ListDir: `target_directory` is empty or only whitespace, and the call passed "{value}"``), then compile every `ignore_globs` entry with `Matcher::new` (a bad one is an `InvalidArguments` error), then `access.stat(target_directory)`. +- Stat failures: `NotFound` gives `Path does not exist: {path}` through `refusal::missing_root` (add it now). Any other stat failure goes through `refusal(...)`. A `Directory` or `Symlink` goes on to the walk (a link named as the target is walked through, and if listing it fails with `NotADirectory` the answer is `Path is not a directory: {path}`). Any other file type gives `Path is not a directory: {path}`. Failures are `ToolError::message(...)`, `path` is as the model wrote it. +- Output is a tree, lines joined with `\n` and no trailing newline. The root line is the absolute virtual path plus `/`: the target as written, with a leading `/` added when it lacks one (no canonical path is available), and `/` appended unless it already ends in `/` (root `/` stays `/`). Each child is indented two spaces per depth with a `- ` bullet and a trailing `/` on directories only. Files and directories share one list in byte order of the name with `/` appended to a directory's name (the Pack sorts each listing itself, because `Access::list` promises no order). A link is listed by its name with no trailing `/` and is never entered. Entries named `.git`, file or directory, are skipped at any depth. Every other dotfile is listed, so `.env` and `.gitignore` appear. An entry that matches an `ignore_globs` pattern is left out, and a matching directory is not entered; match against the entry's absolute virtual path (the root as written plus the names), with a directory's path having no trailing `/`. A root with no children to show prints `{root line}` then `\n... no children found ...`, with no indent, for example `/home/dev/proj/empty/` then `... no children found ...`. +- The walk is breadth first, so it stops at the byte cap instead of walking a whole tree first. Count the bytes of every tree line, each with its newline, against `BODY_CAP`. When the next line would pass the cap, stop everything: the directory being listed keeps the children already added, and every directory shown but not expanded ends in `...` (for example ` - src/...`). Then the output ends with `\n ...` (four spaces, then three dots), a blank line, and `target_directory is too large (either too many children or their names are too long). Immediate children were truncated. Try a more specific target_directory or ignore_globs.` These two do not count against the cap. Printing walks the finished tree depth first in the sorted order. +- Every successful ListDir returns `ToolOutput::untrusted`. Dropped from Cursor's `LS`: extension-count summaries, the 5-second timeout, the terminal-log special case, and the `Error listing directory:` prefix. + +Definition and wiring: + +- New `schemas/dir.json`: `wire_name` `ListDir`; parameters `target_directory` (string, the absolute path of the directory to list) and `ignore_globs` (array of strings, globs of entries to leave out; a pattern with no leading `/` matches at any depth, and only `*` and `**` are special); `required` is `["target_directory"]`. The description says: a bullet tree of the directory, directories end in `/`, files and directories share one byte-ordered list, the walk is breadth first, output stops at `{body_cap}` bytes with a note, entries named `.git` are skipped and every other dotfile is listed, links are listed and not entered, and `ignore_globs` match at any depth. Fill `{body_cap}` from `crate::text::BODY_CAP` with the fill function in `spec.rs`. The description names no PDFs, no `Shell` tool, and no Cursor workspace. +- `schemas/EDITS.md`: add a `dir.json` section that says this file is new, is not a Cursor snapshot, and why (Cursor's counterpart `LS` is deprecated and gated behind a setting, so the wire name is `ListDir`). +- `src/spec.rs`: `SOURCES` becomes seven entries with `("dir", include_str!("../schemas/dir.json"))` last; update its module doc ("seven"). `src/pack.rs`: add `"dir"` to the dispatch. `lib.rs`: add `mod dir;` and `mod matcher;`, add `| std/dir | ListDir |` to the table in the crate docs, say seven tools, and say Read and ListDir are real while Write, StrReplace, Delete, and Glob are stubs. +- Move the tool-count tests from six to seven: `the_specs_are_the_six_the_crate_names_in_order` in `pack-tests.rs` becomes seven with `["read", "write", "replace", "rm", "glob", "grep", "dir"]`; `WIRE_NAMES` in `spec-tests.rs` becomes `[(&str, &str); 7]` with `("dir", "ListDir")` last; in `tests/it/tools.rs`, `construct_names_six_tools_under_the_pack_id_it_receives` becomes seven and lists `files/dir` last, and the wire-name test adds `("std/dir", Some("ListDir"))` last. Update the `tools.rs` module doc that says six. + +Tests: + +- Unit, `dir/tests.rs` and topic files: + - The POSIX tree. Seed `/home/dev/proj` with `README.md`, `package.json`, `docs/guide.md`, `src/app.tsx`, `src/index.ts`, and `src/utils/math.ts`, call with `{"target_directory": "/home/dev/proj"}`, and assert exactly this text (the parent report `runvil-design/specs/cursor/agent-file-tools.md` shows `package.json` before `README.md` because Cursor sorts with `localeCompare`; this plan's byte order governs, so `README.md` comes first): + ```text + /home/dev/proj/ + - README.md + - docs/ + - guide.md + - package.json + - src/ + - app.tsx + - index.ts + - utils/ + - math.ts + ``` + - An empty directory (`/home/dev/proj/empty/` then `... no children found ...`), a root holding only `.git`, a file as the target (`Path is not a directory: {path}`), a missing path (`Path does not exist: {path}`), and a blank target. + - `.git` skipped while `.env` and `.gitignore` are listed, and a `.git` that is a file skipped too. + - Sort order against `Access::glob`'s for the same tree: seed `/B`, `/a-b/x`, `/a.txt`, `/a/x`, `/b` and assert the tree lists `B`, `a-b/`, `a.txt`, `a/`, `b` in that order, the same order `access.glob("/**")` gives for the files (`/B`, `/a-b/x`, `/a.txt`, `/a/x`, `/b`). + - A `Symlink` entry through the lister parameter is listed by name with no `/` and is never entered. + - The byte cap and its note: lower the cap, assert the `...` suffix on unexpanded directories, the `\n ...` line, the blank line, and the too-large sentence byte for byte; sweep the cap over a tree of multi-byte names and assert the output is valid UTF-8 and every cut sits on a line boundary. + - `ignore_globs` matching at any depth (`node_modules` and `*.log` leave out entries at every level, `/build` only the one at the root), and a refused pattern (`a***`, `a/**b`, an empty string) as an `InvalidArguments` error with the text above. + - A trust test: every successful ListDir is `OutputTrust::Untrusted`. + - The description pin, in `spec-tests.rs`: the built `dir` description contains `15000` and no unfilled `{body_cap}`. +- Integration, `tests/it/tools.rs`: add a ListDir test that seeds a small tree through the call's access and asserts the tree text as untrusted output. + +Commit: one commit holding the matcher, ListDir, `dir.json`, `EDITS.md`, `spec.rs`, the dispatch, the doc updates, the six-to-seven test edits, and every test. Suggested subject: `Add the std glob matcher and a real ListDir`. + + + + + +### Step 4: Real Grep, the real-directory test, and the full gate set [completed] + +- Component: `grep-tool` +- Tests: + - `cargo nextest run --locked -p plugin-std` + - `cargo nextest run --locked --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-features` + - `cargo nextest run --locked -p workshop -p workshop-server -p workshop-server-api` + - `cargo clippy --workspace --exclude workshop --exclude workshop-server --exclude workshop-server-api --all-targets --all-features` + - `cargo clippy -p workshop -p workshop-server -p workshop-server-api --all-targets` + - `cargo fmt --all --check` + - `RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --all-features --exclude workshop --exclude workshop-server --exclude workshop-server-api` + - `cargo test -p build-xtask` + - `node --test crates/workshop/ui/test/docs-claims.mjs` + - `cargo +nightly-2026-09-05 xtask api --check` + +Run both clippy commands with `CARGO_BUILD_WARNINGS=deny` (PowerShell: `$env:CARGO_BUILD_WARNINGS = "deny"`), run the doc command with `RUSTDOCFLAGS` set to `-D warnings` (PowerShell: `$env:RUSTDOCFLAGS = "-D warnings"`), and run the api check with `CARGO_BUILD_WARNINGS` unset (PowerShell: `Remove-Item Env:CARGO_BUILD_WARNINGS`). The nightly name must match `PINNED` in `crates/build-xtask/src/api/toolchain.rs`. + +Placement: last, because Grep needs the text module and refusal text from step 2 and the matcher from step 3, and because the full gate set and the real-directory test need all three tools real. The component has three pieces. The Grep tool and its yield are built jointly, because the walk and the yield are one loop and the yield is tested by polling that same future. The real-directory test comes after them in sequence, because it drives ListDir, Grep, and Read together. The full gate set runs last and replaces a separate gates step. This is the largest step and it stays one commit, per the four-step decision. + +Rules for every step in this plan: warnings are denied and clippy runs `pedantic`, so add no item before something uses it, give every item a doc comment, and put `#[expect(..., reason = "...")]` on the casts that turn JSON numbers into `usize`. Tests may use `unwrap` and `expect`. New code adds no `#[path]` attribute: a module is `foo.rs` beside a `foo/` directory, with its unit tests in `foo/tests.rs`. Every Rust file stays at or under 500 lines. Do not touch the Engine, the Harness, the vfs crate (this step changes no file in it), or any file under `vibe/`. Do not make `plugin-std` depend on the vfs crate: import `Access`, `VfsError`, `Entry`, `FileType`, and `Stat` from `runvil_plugin`. + +Code from steps 2 and 3 that this step uses, already in the tree: `src/refusal.rs` (`refusal(tool, path, &VfsError) -> String`, `missing_root`), `src/check.rs` (`check`, `refuse_unknown`, the blank-path helper, `plural(n, noun)`, and the matcher-refusal text builder), `src/text.rs` (`decode`, `sniff_binary`, `split_lines`, `clip_to_boundary(text, start, end)`, `MAX_FILE_BYTES`, `BODY_CAP`), `src/matcher.rs` (`Matcher::new`, `Matcher::is_match`), and the fill function in `spec.rs` that replaces `{name}` placeholders in a description with numbers from the code's constants. The shapes this step relies on: the unknown-key text is ``std_{Wire}: `{key}` is not a parameter of this tool. The parameters are `a`, `b` and `c`.`` with the names sorted bytewise; the blank-path text is ``std_{Wire}: `{name}` is empty or only whitespace, and the call passed "{value}"``; `plural(n, noun)` gives `1 line` and `10 lines`, and `directory` becomes `directories`. + +Definition, in `crates/plugin-std/schemas/grep.json`: + +- Replace the property set with the bundle schema plus `literal`, in this order: `pattern` (string, required), `path`, `glob`, `output_mode`, `-B`, `-A`, `-C`, `-i`, `type`, `head_limit`, `offset`, `multiline`, `literal`. Types: strings for `pattern`, `path`, `glob`, `type`; `output_mode` is a string enum `["content", "files_with_matches", "count"]` in that order, default `content`; `-B`, `-A`, `-C`, `head_limit`, and `offset` are `number`; `-i`, `multiline`, and `literal` are `boolean`. `-n` and `context` are deleted. `-C` takes the text that was on `context`: `Number of lines to show before and after each match. Requires output_mode: "content", ignored otherwise.` `head_limit` has no default, `"minimum": 0`, and its text says `0` means no limit. `path` reads `File or directory to search in, as an absolute path. Defaults to the run's filesystem root.` `glob` reads `Glob pattern to filter files, for example "*.js" or "**/*.tsx". Only * and ** are special, and matching is case-sensitive. A pattern that does not start with "/" is matched at any depth. A pattern that starts with "/" is a full path.` `literal` is a boolean, default false, whose text says the pattern is matched as plain text. Keep Cursor's wording for the other parameters, with `output_mode` now defaulting to `content`. Keep the hyphenated names and add no aliases. The existing integration tests for a missing `pattern` and for `output_mode: "lines"` must still pass with the same texts. +- Description: name the dialect (`Rust regex` syntax, with no look-around and no back-references; escape `( [ { .` or set `literal`), the order (path order, the order Glob uses), the limits (output stops at `{body_cap}` bytes, filled from `crate::text::BODY_CAP`; at most 100,000 files are considered; a file over `{file_ceiling}` bytes is not searched, filled from `crate::text::MAX_FILE_BYTES`), what is skipped (binary files, symbolic links, over-size files, and unreadable entries are skipped and the result says how many were skipped and why), the ignore rule (only entries named `.git` are skipped and `.gitignore` rules are not applied, so output such as `target/` and `node_modules/` is searched when it lies under `path`), that a line over 1000 bytes is shown as a window around its first match, that paths are absolute and work as the `path` argument of the other file tools, that `N:` and `N-` are not part of the line and `N` is the line number Read shows, and that the model should read the file before changing it. The text does not contain `ripgrep`, `rg ` (check that no word ends in `rg` before a space), `several thousand`, or `Shell`. Fill the two placeholders with the fill function in `spec.rs`. Record every change in `schemas/EDITS.md`: the new bundle parameters, the deleted `-n` and `context`, the new `literal`, the `content` default, the new description, and that the `rg` and `Shell` removals from the earlier snapshot still hold. + +Arguments, in `src/grep/args.rs`. `null` reads as absent and numeric strings are not coerced (the shared `check` does both). An empty `path`, `glob`, or `type` reads as absent, and a whitespace-only `path` is refused with the blank-path text. Every error below is a `ToolError` of kind `InvalidArguments`. Check in this order and stop at the first failure: + +1. `check::check`, then `refuse_unknown` run on a copy of the arguments with `-n` removed (so `-n` is the one undeclared key that is accepted, and the shared helper stays unchanged). The refusal names the key and lists the thirteen parameters, sorted, in the shape above. +2. `-n`: `true` or absent is accepted, and any other value is refused with ``std_Grep: `-n` must be true or left out, because line numbers are always shown.`` +3. Empty pattern: ``std_Grep: `pattern` is empty. Pass "." to match every line.`` +4. A negative `-A`, `-B`, or `-C`, checked in that order: ``std_Grep: `-A` is -1. Pass 0 or more context lines.`` with the key and the passed value filled in. +5. A negative `head_limit` or `offset`, in that order: ``std_Grep: `head_limit` is -5. Pass 0 or more.`` +6. An unknown `type`: ``std_Grep: Unrecognized file type "foo". Use a type such as js, ts, py, rust, go or java, or filter with glob.`` +7. An invalid `glob` (`Matcher::new` fails), using the shared matcher-refusal text builder in `check.rs` with `{name}` set to `glob`: ``std_Grep: `glob` is invalid: "{glob}" does not parse: {reason}.`` +8. A whitespace-only `path`: ``std_Grep: `path` is empty or only whitespace, and the call passed " "``. +9. The pattern compile (below), then the `path` stat. + +Rounding: `head_limit` and the context values round up, and `offset` rounds down. Absent `head_limit` and `head_limit: 0` both mean no limit. Effective context: `-C` sets both sides, then `-A` sets the after side and `-B` the before side, each overriding `-C` even when it is 0. Context applies only in `content` mode and is ignored in the other two modes. + +Pattern compile, before any filesystem call, with `regex::RegexBuilder`: `case_insensitive` from `-i`, `multi_line(true)` always, and `dot_matches_new_line(true)` and `crlf(true)` only with `multiline`. `literal` compiles `regex::escape(pattern)`. A failure returns ``std_Grep: Invalid regular expression "": . Grep matches with Rust regex syntax, which has no look-around and no back-references. To match the text literally, set literal to true.`` where `` is the text after the first `error: ` in the crate's message, or the whole message trimmed when it has none. Outside `multiline` matching is line by line; with it the whole decoded text is one subject. The default is case-sensitive. + +Type table, in `src/grep/types.rs`, matched by file-name extension (the text after the last `.` when that dot is not the first character), case-sensitive: `js` (js, mjs, cjs, jsx), `ts` (ts, tsx, mts, cts), `py` (py, pyi), `rust` (rs), `go` (go), `java` (java), `c` (c, h), `cpp` (cc, cpp, cxx, hh, hpp, hxx, h), `cs` (cs), `rb` (rb), `php` (php), `md` (md, markdown), `json` (json), `yaml` (yaml, yml), `toml` (toml), `html` (html, htm). + +Search, in `src/grep/find.rs`. `find` is a pure async function over a small `Source` trait that `Access` implements through a thin adapter (`stat`, `list`, `read`, all synchronous), and returns what it found; `render` turns that into text. The seam exists because the in-memory backend cannot produce a symbolic link, a control-character name, an over-size entry, a denied read, or a conflict, and the tests need them. `Limits` is a struct of parameters with the constants as defaults: `max_files` 100,000, `max_bytes` 1 GiB, `max_file_size` 16,777,216, `slice_files` 32, `slice_bytes` 2 MiB, and the output cap `BODY_CAP`; tests lower each. + +- Root: `path` defaults to `/`. A root without a leading `/` is made absolute by adding one. `Access::stat` runs first. `NotFound` fails the call with `Path does not exist: {path}` through `refusal::missing_root`, and any other stat failure fails through `refusal(...)`, with `path` as the model wrote it. A `Directory` is walked, and a failure to list the root itself fails the call through `refusal(...)`. A `File` is searched alone, with no ignore rule, and `glob` and `type` do not filter it, because the model named it. A `Symlink` is tried as a directory and then as a file, so a `path` that names a link is searched through it. +- Order: depth first, each directory's entries in byte order of the name with `/` appended to a directory's name (sorted by the Pack, same order as `Access::glob`). For example the files `/B`, `/a-b`, `/a.txt`, `/a/x`, `/b` come in exactly that order. No clock is read. Result lines print the absolute virtual path with its leading `/`: the root as written plus the names. +- Entries inside the walk: an entry named `.git` is skipped silently; any entry whose name has a character below 0x20 or 0x7f is a counted skip (`name with a control character`) and its name is never printed; a `Symlink` entry is a counted skip (`symbolic link`) and is never entered or read; a `Directory` is entered; every other entry is a file. For each file: count it as considered (when a 100,001st file would be considered, stop with the files notice below), apply `glob` (the step-3 matcher on the absolute path) and `type` (a file that fails a filter is neither searched nor a skip), then skip it as `over 16 MiB` when its size in the directory listing is above 16,777,216 bytes, decided before any read. Then read it with `Access::read` (never `read_string`), `sniff_binary` (a binary file is a counted skip `binary`, in every output mode), `decode`, and `split_lines`, so a Grep line number names the line Read prints. Searching a file counts it as searched. +- No single entry fails the search. Skip reasons and how a `VfsError` maps: `Conflict` is `another call in this round`; `PermissionDenied` and `ApprovalRequired` are `denied by policy`; any other listing failure is `unreadable directory`; any other read failure is `unreadable file` (a reason this plan adds to the seven above, so a failed read is never silent). Each skip is counted by reason, and the first five printable skips are kept in walk order for the skip line. +- Limits: stop at 100,000 files considered, or when another file would be read after 1 GiB has been read. A limit stop prints its notice and offers no `offset`, because a re-run stops at the same file. +- Yield: after every 32 files considered or 2 MiB read since the last yield, `.await` a hand-written `Pause` future in `src/grep/pause.rs` (a struct whose `poll` wakes its own task through `cx.waker().wake_by_ref()` and returns `Pending` once, then `Ready`; name no async runtime; do not call the module `yield`, which is a reserved word). Read and ListDir do not yield. A cancel or stop drops the future at its next await point. +- Collecting: counting is global across files and in entries (a matching line in `content` mode, a file in `files_with_matches`, a file row in `count`), and context lines are never entries. Skip the first `offset` entries, collect `head_limit` more plus one extra entry that only decides whether a notice is due, then stop the walk; with no head limit collect all. Also stop collecting once the entries after `offset` already fill the body cap and one more exists, because entries past that point are never shown. Keep memory bounded: store only the lines a kept entry shows (the match line as its windowed text, and its context lines clipped), never a whole decoded file after it is processed. In `multiline` mode every line a match touches is a match line and one entry (a match covers the lines from the one holding its first byte to the one holding its last byte; an empty match touches the line it starts on); map byte offsets to line numbers over the decoded text, and strip the one trailing `\r` only from the displayed text. + +Render, in `src/grep/render.rs` and `src/grep/window.rs`. There is no envelope. + +- `content` mode: each file's absolute virtual path appears once, then its lines as two spaces, the line number with no padding, `:` for a match or `-` for context, and the text (` N:text`, ` N-text`). A blank line separates files and none ends the output. Between two non-adjacent groups of one file print a gap line, two spaces then two hyphens, whenever the effective before or after context is above 0. Overlapping or touching context merges and no line prints twice. `files_with_matches` prints one path per line, and `count` prints `path:count` with the matching-line count of that file (the model reads the count after the last colon). Worked example for `const` with `-A: 1`, where `/src/a.ts` matches on lines 3 and 9 of 10 and `/src/b.ts` matches on line 1: + ```text + /src/a.ts + 3:const alpha = 1 + 4-let beta = 2 + -- + 9:const gamma = 3 + 10-export default gamma + + /src/b.ts + 1:const first = 0 + ``` +- Long lines, over 1,000 bytes. A matching line becomes a window of at most 1,000 bytes that contains the first match and starts up to 200 bytes before it: `start = first_match_start - 200` (not below 0), then `clip_to_boundary(text, start, start + 1000)`. Put `[... {n} bytes omitted]` directly before the window when its start is above 0 and directly after it when its end is short of the line, so the two counts plus the window equal the line's byte length. In `multiline` mode the first match on a line is the first byte of the line a match covers. A context line over 1,000 bytes keeps `clip_to_boundary(text, 0, 1000)` and a trailing `[... {n} bytes omitted]`. These two cuts, in `window.rs`, are the only cuts inside a line and both go through `clip_to_boundary`, so no cut splits a code point. The 1,000-byte figure and the 200-byte lead are parameters with these defaults. +- Output cap: 15,000 bytes of body in every mode (`BODY_CAP`), counting each line with its newline, cut on a line boundary: stop before the line that would pass it. The paging, skip, note, and limit lines described below are not counted. The cap is a parameter, so tests can lower it. + +Output assembly, in `src/grep/render.rs`: + +- The text is the body, then, when there is anything to add, a blank line and then the extra lines, each on its own line and joined by `\n`, with no newline at the end. Extra lines, in this order: the replacement note, the skip line, then the paging or limit notice. A zero-match result puts `No matches found. Searched {plural(n, "file")}.` where the body would be (for example `No matches found. Searched 412 files.` and `No matches found. Searched 1 file.`), followed by the extras in this order: the replacement note, the skip line, the line `[No ignore rules ran: .gitignore is not applied, and only entries named .git are skipped.]`, and, when `glob` holds `{`, `[`, or `?`, the line `{, [ and ? in glob match themselves; there is no alternation or character class. Call Grep once per alternative, or filter with type.` +- Replacement note: for each searched file that needed a replacement, `[Note: invalid UTF-8 in {plural(n, "file")} was shown as U+FFFD: {path}, {path}]`, which reads `invalid UTF-8 in 2 files was shown as U+FFFD: /a.txt, /b.txt` and `invalid UTF-8 in 1 file was shown as U+FFFD: /a.txt`. Name the first five paths and end the list with `, and {n - 5} more` when there are more. +- Skip line: `[Skipped {plural(total, "entry")}: {phrases}. First: {paths}]`, for example `[Skipped 5 entries: 3 binary files, 1 directory that could not be read, 1 file over 16777216 bytes. First: /a.bin, /b.bin, /c.bin, /d, /big.log]` and `[Skipped 1 entry: 1 binary file. First: /a.bin]`. The phrases come in this fixed order, each with its count and the `plural` helper: binary (`{n} binary file(s)`), unreadable directory (`{n} director(y/ies) that could not be read`), unreadable file (`{n} file(s) that could not be read`), over 16 MiB (`{n} file(s) over 16777216 bytes`), symbolic link (`{n} symbolic link(s)`), name with a control character (`{n} name(s) with a control character`), another call in this round (`{n} entr(y/ies) held by another call in this round`), and denied by policy (`{n} entr(y/ies) denied by policy`). `First:` lists up to five printable skipped paths in walk order; a control-character name is never printed, and when no printable skip remains the ` First: ...` part is dropped. +- Paging notice. There is no cache, and each call re-runs from the start. The unit word is `matching lines` in `content` mode and `files` in the other two. Print no notice when `offset` is 0 and nothing was cut, so an exact fit prints none. A cut by `head_limit` or by the body cap ends with `[Showing {offset + 1}-{offset + shown} of at least {offset + collected} {unit}. Use offset={offset + shown} to see more, or narrow the search.]`, where `shown` is the entries printed and `collected` is the entries gathered including the extra one; for example `[Showing 1-50 of at least 51 matching lines. Use offset=50 to see more, or narrow the search.]`. When the body cap caused the cut, insert `Output stopped at 15000 bytes.` after the `of at least` clause: `[Showing 1-120 of at least 121 matching lines. Output stopped at 15000 bytes. Use offset=120 to see more, or narrow the search.]`. A page that reaches the end after the first page shows an exact total, `[Showing 51-60 of 60 matching lines.]`. An `offset` past the end prints only `No more results at offset={offset}.` A limit stop prints `[Search stopped after considering 100000 files. Narrow path, glob or type to search the rest.]` or `[Search stopped after reading 1073741824 bytes. Narrow path, glob or type to search the rest.]` and offers no `offset`. +- Trust: every Grep result is `ToolOutput::untrusted`, except a zero-match result that has no replacement note and no skip line, which holds only fixed text and is `ToolOutput::trusted`. `No more results at offset=N.` is untrusted. + +Files for Grep: `src/grep.rs` (entry: `pub(crate) async fn call(access: &Access, definition: &Definition, args: &Value) -> Result`), and under `src/grep/`: `args.rs`, `types.rs`, `find.rs`, `pause.rs`, `render.rs`, `window.rs`, and `tests.rs` with a `tests/` directory split by topic (arguments, find and skips, render, window, paging) so no file passes 500 lines. + +Wiring and docs: + +- `src/pack.rs`: `"grep"` dispatches to `crate::grep::call(cx.access(), definition, &args).await`. Only Write, StrReplace, Delete, and Glob keep the stub. `lib.rs`: add `mod grep;`, and update the crate docs: Read, ListDir, and Grep are real, the other four are stubs that answer `NOT_IMPLEMENTED`, and the invariant about undeclared keys now says the three real tools refuse them by name while the stubs let them through. +- `tests/it/tools.rs`: remove `("grep", "Grep")` from `every_tool_answers_the_not_implemented_text_naming_itself`, leaving the four stubs (`write`, `replace`, `rm`, `glob`) with their stub tests. Add a Grep test that seeds `/src/a.ts` and asserts the `content` output above as untrusted text, and keep the missing-`pattern` and bad-`output_mode` tests as they are. Add new integration modules (for example `tests/it/grep.rs`, declared in `tests/it/main.rs`) if `tools.rs` nears 500 lines. + +Tests: + +- Unit, arguments: every string above byte for byte, in order, the empty-string rules (`path: ""`, `glob: ""`, and `type: ""` read as absent; `path: " "` refused), undeclared-key refusals (`context: 3` is refused naming `context` and listing the sorted parameters; `-n: false` is refused with the `-n` text; `-n: true` is accepted), the rounding (`head_limit` 2.2 gives 3, `offset` 2.9 gives 2), and the `-C` with `-A` and `-B` override. +- Unit, regex: send `(?=x)y`, `(?<=a)b`, `(a)\1`, and `foo(` and assert a `std_Grep: Invalid regular expression "": ` error for each that names the reason and the Rust dialect, as an error and not as success text. `a.b` with `literal: true` over files holding `axb` and `a.b` gives one match. `^(a|aa)+$` over a 1,600,000-character line completes. `multiline` with `a.b` over `a\nb` matches, and without `multiline` it does not. `alpha` over `Alpha`, `alpha`, `ALPHA` gives one match, and three with `-i`. +- Unit, search and order: full-path order on `/B`, `/a-b`, `/a.txt`, `/a/x`, `/b` in all three modes; `head_limit` global across files (`/a.ts` with matches on lines 12 and 40 and `/b.ts` with three: `head_limit: 2` shows only `/a.ts` with both lines and a notice saying `at least 3`; three files of three matches with `offset: 4, head_limit: 2` return the 2nd and 3rd matches of the second file; `head_limit: 0` with 40 matches prints all 40; `head_limit: 2` with exactly two matches prints no notice; `-A: 5, head_limit: 2` gives two entries and their context lines). +- Unit, paging: seed 25 matches and run `head_limit: 10` at offsets 0, 10, and 20: the three pages joined equal the unpaged result, and the notices are `[Showing 1-10 of at least 11 matching lines. Use offset=10 to see more, or narrow the search.]`, `[Showing 11-20 of at least 21 matching lines. Use offset=20 to see more, or narrow the search.]`, and `[Showing 21-25 of 25 matching lines.]`. Offset 30 gives `No more results at offset=30.` A write to a seeded file between two page calls shows in page 2 (no cache). A lowered file limit gives the files notice with no `offset`. +- Unit, render: the worked example above; `-C: 3, -A: 1` with matches at lines 5 and 20 of a 30-line file gives lines 2-6 and 17-21; `-C: 2` with matches at lines 5 and 7 gives lines 3-9 once with no gap line; no context flag gives no gap line; the last 30 bytes of a two-file result end at `const first = 0` with no trailing newline; `count` mode prints `path:count` (also for a file name that holds a colon, if the memory backend takes the name); the byte cap with 400 matching lines of 100 bytes gives a body of at most 15,000 bytes and the notice with `Output stopped at 15000 bytes.`. +- Unit, windows: a 2,000-character line with `NEEDLE` at position 1,001 shows `NEEDLE`, both markers, and omitted counts that plus the window equal the line's byte length; a line of 600 CJK characters, and a line of 4-byte emoji with the match in the middle, give valid UTF-8 and whole characters at both ends; a context line of 600 CJK characters is clipped to at most 1,000 bytes of valid UTF-8; sweep the window's start over every byte position inside one 3-byte character, and sweep the body cap and the window sizes over emoji and CJK text, asserting valid UTF-8 and line-boundary cuts everywhere. +- Unit, skips and limits, through the `Source` seam with lowered limits: a binary file, an over-size file, a denied directory (`PermissionDenied` on list), a denied file, a `Conflict` on a file, a symbolic link, a control-character name, an unreadable file (an `Io` error), and one clean match. Assert the clean match, the skip line with the counts and the first five printable paths in walk order, the zero-match line with the same skip line and the right searched count (`Searched 1 file.` for one), and that a missing `path` fails with `Path does not exist: /x`. Reach each limit (files considered, bytes read, per-file size, slice size) by lowering its parameter. A `[Skipped 1 entry: 1 binary file. First: /a.bin]` case. +- Unit, text (through the shared `text` module, so a Grep line number is the number Read prints): `caf\xE9 NEEDLE` matches on line 1 with U+FFFD and the replacement note; a file that starts with a byte order mark and `NEEDLE first` matches `^NEEDLE` on line 1 with no mark in the output; `a\r\nNEEDLE\r\n` matches on line 2 with no `\r`; `first\rNEEDLE\rthird` is one match on line 1; UTF-16LE with a mark and `NEEDLE` on line 2 matches on line 2; UTF-16LE without a mark is one binary skip; 100 bytes with 5 control bytes and no NUL is a binary skip and 4 control bytes is a match; `é` as one code point and as two each match only their own file; two replaced files give `[Note: invalid UTF-8 in 2 files was shown as U+FFFD: /a.txt, /b.txt]`. +- Unit, filters: `glob: "*.js"` matches at any depth, `glob: "/src/**/*.rs"` is a full path, `type: "rust"` keeps `.rs` files, a named single file ignores both filters, and a zero-match search with `glob: "*.{ts,tsx}"` adds the hint line. +- Unit, trust: results are `OutputTrust::Untrusted`, and the bare zero-match result is `OutputTrust::Trusted`. +- Description pin, in `spec-tests.rs`: the built `grep` description contains `Rust regex`, `literal`, `15000`, `16777216`, and `skipped`, and does not contain `ripgrep`, `rg `, `several thousand`, or `Shell`; no placeholder is left unfilled; the `glob` parameter text holds `"*.js"` and `"**/*.tsx"` and no `{`; the `path` text says `absolute`. +- Yield check, in the integration target: seed 1,000 small files through the call context's `Access` and poll the Grep future by hand with a no-op waker, counting `Pending` results. Assert at least 31 `Pending` before it completes with the right result, and that dropping a second copy of the future after its first `Pending` is safe (nothing hangs or panics and the access still works). `TestCall` cannot supply a chosen `VfsRef` or an event sink, so the test counts `Pending` polls and not reads. +- Real directory: one test over a directory mounted with `RealBackend::rooted`. It cannot live in `plugin-std`, so put it in `crates/workshop/server/tests/it/` as a new module (for example `std_real_directory.rs`, declared in `tests/it/main.rs`): that crate already depends on `plugin-std`, `runvil`, `runvil-plugin`, and `tempfile`. Use another existing suite only if this one cannot build, and keep the crate allowed to depend on both `plugin-std` and the vfs crate (`cargo test -p build-xtask` checks it). Build the context directly with `ToolContext::new(&tool, &access, &origin, &services)`, using `VfsRef::builder().mount("/", RealBackend::rooted(dir)?).build()`, `vfs.acquire(Origin::new(...))`, `runvil::effect::ToolCallOrigin` and `ToolCaller::Script`, and `HostServices::new()`, and drive the Pack that `plugin_std::install` inserts. Name the temp directory from the test name, the process id, and a counter. Seed a file whose 5,000 lines are 40 `x` each except line 3,000, which holds a unique word. Then: ListDir on `/` lists the file; Grep for the word returns that file and `3000:` with the line; Read of the file with no arguments returns a first page ending with a trailer that names `offset=2084`; Read with `offset: 2084` starts at ` 2084|` (six-wide number, pipe, no space) and continues. This is the acceptance flow: list, find, read a window, read the next window by the trailer offset. +- Regression: the four stub tools still return the stub text unchanged, and every existing `plugin-std` test that this plan did not change passes. + +Exit: the whole gate set in Tests passes, including the Workshop crates, both clippy runs, the rustdoc run, `cargo test -p build-xtask`, the wording scan, and the facade check (the `Access` additions from step 1 change the facade listing, which step 1 regenerated). If a gate fails for a reason outside this work, stop and report it instead of editing unrelated files, except for edits this plan makes (`Cargo.lock`, the docs and test updates above). + +Commit: one commit holding Grep, `grep.json`, `EDITS.md`, the dispatch, the doc updates, the yield and real-directory tests, and every other test. Suggested subject: `Add a real Grep and a real-directory test for the std file tools`. + + + +