diff --git a/CHANGELOG.md b/CHANGELOG.md index fbb5628..3d28122 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,9 +6,20 @@ (`jacquard-host`, Rust 1.88+). It provides strict codecs, the session state machine, a bounded process carrier for `jacquard host worker`, and a `Host` with a closed operation registry. It also embeds the pinned kit and replays - all 21 recordings. It is protocol-level only, with no SDK layer. - `scripts/core_pin.py` now generates `rust/src/compat.rs` and keeps - `rust/kit/` in sync, and CI runs the crate's offline and live tests. + all 21 recordings. `scripts/core_pin.py` now generates `rust/src/compat.rs` + and keeps `rust/kit/` in sync, and CI runs the crate's offline and live + tests. +- Rust SDK (`jacquard_host::sdk`): prepares and caches checked programs with + Core's public commands, maps native Rust values through the `Encode` and + `Decode` traits (with `record!` and `variants!` for application types, and + `Dynamic` for untyped code), binds closures to exact operation identities, + checks arguments against the checked signature before a worker starts, and + keeps each invocation's raw evidence. `map_batches` runs explicit batches. + The four example projects run as `cargo run --example jacquard-examples`. + The low-level host gains `From for HandlerError`, so `?` in a + handler reports completion unknown. As a consequence `HandlerError` cannot + implement `std::error::Error`, and a downstream `From` impl for a type that + implements `Error` conflicts with it. ## 0.2.0 — 2026-10-25 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1c0bde7..f1f5727 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -90,9 +90,9 @@ a file under `examples/`, copy it there too (a Go test fails if they differ). ## Rust -The independent Rust adapter lives in `rust/` as crate `jacquard-host`. It is -protocol-level only, with no SDK layer. It uses only the standard library and -does not call Python, Node.js, or Go. Rust 1.88+ is required. +The independent Rust adapter and SDK live in `rust/` as crate `jacquard-host`. +It uses only the standard library and does not call Python, Node.js, or Go. +Rust 1.88+ is required. ```sh cd rust @@ -107,7 +107,9 @@ CI runs fmt, clippy, and the offline tests on Rust 1.88 and the current stable release. It runs the live tests in the shared pinned-Core job. `rust/src/compat.rs` is generated and `rust/kit/` is a byte copy of the pinned kit. `scripts/core_pin.py` maintains both; do not edit them by hand. The -`jacquard-host-standin` binary is a test fixture for the offline tests. +`jacquard-host-standin` binary is a test fixture for the offline tests. The +`jacquard-examples` Cargo example reads the rules and inputs from `examples/` +directly, so there is no copy to keep in sync. ## Releasing diff --git a/README.md b/README.md index 8c394f3..538ffa3 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,17 @@ # Jacquard Host -Call checked Jacquard functions from Python, TypeScript, or Go and implement their +Call checked Jacquard functions from Python, TypeScript, Go, or Rust and implement their declared host operations with ordinary services. Jacquard owns the decision logic; the host owns persistence, I/O, and application lifetimes. The SDK prepares and caches checked programs, converts native Python values, binds services to exact operation identities, and retains the raw evidence for -every invocation. Python 3.11+, Node.js 22+, or Go 1.24+, with no third-party -runtime dependencies. The [TypeScript SDK](typescript/README.md) and the -[Go SDK](go/README.md) implement the pinned protocol independently; the -TypeScript SDK supports async service methods, and the Go SDK maps native Go -types and context cancellation. A protocol-level [Rust adapter](rust/README.md) (Rust 1.88+) -also implements the pinned protocol independently. It has a closed operation -registry and kit replay, but no SDK layer yet. +every invocation. Python 3.11+, Node.js 22+, Go 1.24+, or Rust 1.88+, with no +third-party runtime dependencies. The [TypeScript SDK](typescript/README.md), +the [Go SDK](go/README.md), and the [Rust SDK](rust/README.md) implement the +pinned protocol independently. The TypeScript SDK supports async service +methods, the Go SDK maps native Go types and context cancellation, and the Rust +SDK maps native Rust types through `record!` and `variants!` declarations. **Experimental:** the SDK targets Core's pinned `jacquard-host-v0` protocol. Core is installed separately. Each invocation runs one fresh, serial worker; diff --git a/docs/EVIDENCE.md b/docs/EVIDENCE.md index d62c703..a963f20 100644 --- a/docs/EVIDENCE.md +++ b/docs/EVIDENCE.md @@ -79,6 +79,19 @@ on Linux 7.0. The crate uses only the standard library. | Lint | `cargo fmt --check` and `cargo clippy --all-targets -- -D warnings` are clean; clippy is also clean for `aarch64-apple-darwin` and `x86_64-pc-windows-gnu` (type-checked only, not run) | | Pin consistency | `rust/src/compat.rs` is generated, and `rust/kit/` matches the pinned digests and the Python copy (`scripts/core_pin.py check`) | +## Rust SDK and projects (2026-10-26) + +Recorded against the same `jacquard-core-0.3.0` release asset, with Rust 1.99.1 +on Linux 7.0. Offline tests and clippy also pass on Rust 1.88. The crate still +uses only the standard library. + +| Lane | Result | +|---|---| +| Rust SDK offline | 13 further unit tests, 1 descriptor and thread test, 1 command exit-status test, and 7 further doctests pass: signature parsing, identity groups, and name spellings; native values through checked types (records with optional fields in lists, Real bits, the full Int range, refused out-of-range integers, integers at a `Real` parameter, and mismatched types reported at their argument path); `variants!` enums and unions, wrong constructors and field counts, nested options, `Dynamic` values, refused builtin constructors by name or through `record!`, and depth refusal; selected text and collection limits on encode and decode; tuple and `&dyn Encode` arguments; closure handlers (context, structured failures, arity, unmappable arguments that never run the handler, plain errors as completion unknown, unrepresentable results, replacement); runtime option and source validation before Core runs, with empty options meaning unset; and, with a fake Core, one version check across four concurrent preparations, prelude discovery once across two sources, a failing `check` reported promptly although a descendant holds Core's output, no staging directory left behind, and a version mismatch refused and checked again. On Linux a Core descendant that left the process group and holds Core's output fails preparation within the grace period, and repeated failures leave the thread and descriptor counts unchanged. A Core output reader that cannot start, at either position, stops the other reader and kills the command at once. The built example exits 2 for usage errors | +| Rust SDK live | 12 further tests pass: quickstart calls, rendering, and a saved owner-only inspection file; triage records, enums, and options over 64 rows in explicit batches against an oracle; late invalid input, over-deep calls, wrong argument types, and wrong argument counts refused before any worker; binding completeness, refused unknown operation names, and pure calls without services; unknown completion once with raw evidence; 25 planning cases matching an independent exhaustive oracle with feasible assignments; warm cache reuse without Core commands, byte-exact source invalidation, corruption refusal; full Int range, Real bits, nested options, and decode overflow retaining the result; spawn failure and cancellation; output overflow without adaptive batching; result-mapping failure retaining the raw result; prelude contents in the cache key and concurrent preparation publishing once | +| Rust projects live | 3 further tests pass: all four `jacquard-examples` commands, including duplicate fulfillment, `--inspect`, explicit delivery, a failed delivery printing no count, and usage errors; the built command exiting 1 with E1613 for a failed publish; and the three fulfillment faults (E1612 after commit, E1613 on publish, E1612 after delivery) recovered explicitly across restart, each receipt delivered once | +| Unchanged lanes | The adapter's 28 offline, descriptor, fallback, NOTICE, and 4 live tests, the 21-recording kit replay, `scripts/core_pin.py check`, the offline Python suite, and `scripts/check_distribution.py` pass | + ## Outcome frame node budget (2026-10-25) Core's `outcome` encodes the evidence's interface types and the result value diff --git a/docs/LIMITS.md b/docs/LIMITS.md index 5cf6b16..4889ae6 100644 --- a/docs/LIMITS.md +++ b/docs/LIMITS.md @@ -15,7 +15,8 @@ also updates this document. | Go handler cancellation is cooperative | At the deadline or on context cancellation the Go host kills the worker's process group (while the worker is unreaped, which Linux guarantees), closes its pipes, and reaps it, cancels the handler's context, and sends no late response; a handler that ignores its context keeps running in its goroutine until it returns. No outside action is stopped or rolled back. | A separately enforced operation lifetime with explicit completion and reconciliation evidence. | | Rust worker descendants that leave the process group | The Rust host stops the worker's process group at expiry. On Linux it also stops the group after exit while a descendant still holds stderr, because `waitid(WNOWAIT)` keeps the leader unreaped. Elsewhere, and for a descendant that left the group, it stops waiting once the grace or the lifetime deadline ends. On Unix its pipe threads then stop within one poll interval and the pipes are closed, but the descendant keeps running. Non-Unix targets only type-check: their pipe I/O cannot be interrupted, so disposal can hang while a descendant holds a pipe, and the offline stand-in, and so the offline suite, is Unix-only. | An OS-level containment mechanism (for example a cgroup or job object) with its own evidence. | | Rust handler cancellation is cooperative | Each Rust handler runs on its own thread. At the deadline or on cancellation the host kills the worker's process group, closes its pipes, reaps it, and sends no late response; `HandlerContext::is_cancelled` turns true, and stays true once the invocation ends. A handler that ignores it keeps running on its thread until it returns. No outside action is stopped or rolled back. | A separately enforced operation lifetime with explicit completion and reconciliation evidence. | -| Rust adapter is protocol-level only | Rust callers pass exact identities, boundary types, and protocol values to a prepared store; there is no Rust SDK for program preparation, native type mapping, or caching. | A Rust SDK with typed-binding and preparation evidence comparable to the other SDKs. | +| Rust SDK field order is declared, not exported | `record!` and `variants!` list fields in the order the Rust code declares; the pinned CLI exports no field types, so a wrong order is not detected locally, and Core refuses it only where the swapped fields' types differ. | Core's structured host interface exporter, so bindings can be generated or checked. | +| Rust SDK preparation is not cancellable | Each Core `--version`, `check`, `hash`, and `run` command runs in its own process group with a 30-second timeout and bounded output; `prepare` takes no `Cancel`. On Linux the group is also stopped when the command exits. A descendant that left the group, or on other platforms any descendant, that keeps Core's output open fails preparation after a one-second grace and keeps running; on Unix the reader threads stop and the pipes close, while on non-Unix targets a reader blocked on such a pipe stays until the descendant closes it. | A cancellable preparation API, once an application needs one. | | Node handler cancellation is cooperative | The Node adapter can expire while awaiting a Promise, signal the handler, and kill the worker; it cannot stop an outside action or preempt JavaScript that blocks the event loop. No late response or automatic retry follows expiration. | A separately enforced operation lifetime with explicit completion and reconciliation evidence. | | Socket host closes without a response on its own failures | A ceiling violation, early disconnect, or failed invocation ends the connection silently; clients see a reset, not an HTTP error. | An owner decision that the host may emit fixed minimal status lines, recorded here with the exact bytes and the rule that they carry no application meaning. | | Request bytes cross as Latin-1 `Text` | v0 has no bytes value; each byte is one codepoint 0..255 and a response codepoint above 255 is refused by the host. | A frozen bytes value in a later protocol version. | diff --git a/examples/README.md b/examples/README.md index a759504..03a0310 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,10 +1,12 @@ # Example projects -The same Jacquard rules run through all three SDKs. The commands below use +The same Jacquard rules run through all four SDKs. The commands below use Python; the [TypeScript walkthrough](../typescript/README.md#run-the-projects) -includes compiled Node.js front ends for all four projects, and the +includes compiled Node.js front ends for all four projects, the [Go walkthrough](../go/README.md#run-the-projects) a Go command with the rules -and inputs embedded. Both include an independent fulfillment recovery example. +and inputs embedded, and the [Rust walkthrough](../rust/README.md#run-the-projects) +a Cargo example that reads them from this directory. Each includes an +independent fulfillment recovery example. Start with [setup](../docs/SETUP.md), then run these from the repository root. They use synthetic inputs, the installed Python SDK, and the pinned Core worker. diff --git a/rust/Cargo.toml b/rust/Cargo.toml index 1f45c3a..cf59b86 100644 --- a/rust/Cargo.toml +++ b/rust/Cargo.toml @@ -3,11 +3,11 @@ name = "jacquard-host" version = "0.2.0" edition = "2024" rust-version = "1.88" -description = "Independent Rust adapter for Jacquard Core's experimental jacquard-host-v0 protocol" +description = "Independent Rust adapter and SDK for Jacquard Core's experimental jacquard-host-v0 protocol" license = "Apache-2.0" repository = "https://github.com/jbwinters/jacquard-host" readme = "README.md" -include = ["src/**", "tests/**", "kit/**", "README.md", "LICENSE", "NOTICE", "Cargo.toml"] +include = ["src/**", "tests/**", "examples/**", "kit/**", "README.md", "LICENSE", "NOTICE", "Cargo.toml"] [dependencies] diff --git a/rust/README.md b/rust/README.md index 5aaf0d6..0a231a6 100644 --- a/rust/README.md +++ b/rust/README.md @@ -1,22 +1,26 @@ # Jacquard Host for Rust -An independent Rust adapter for Jacquard Core's experimental `jacquard-host-v0` -protocol. It implements the pinned protocol on its own, without calling Python, -Node.js, or Go, and uses only the standard library. Rust 1.88+ is required. +Call checked Jacquard functions from Rust, bind Rust closures to their +declared operations, and inspect the exact protocol evidence. This is an +independent Rust implementation of the pinned `jacquard-host-v0` boundary. It +uses only the standard library, and runs Core directly without Python, +Node.js, or Go. Rust 1.88+ is required. -This is the protocol-level adapter, comparable to the Go `host` package. It -includes strict codecs, the host session state machine, and a bounded process -carrier for `jacquard host worker`. It also provides a `Host` with a closed -operation registry and a replayer for Core's pinned conformance kit. There is -no SDK layer yet. It does not compile programs, map native types, or cache -stores. You prepare a store with the Core CLI and pass exact identities and -boundary types. +The crate has two layers: +- **The SDK** (`jacquard_host::sdk`) prepares and caches checked programs, + maps native Rust values, and binds closures to exact operation identities. + It is comparable to the Python, TypeScript, and Go SDKs. +- **The protocol-level host** (`jacquard_host::host`) takes exact identities, + boundary types, and protocol values for a store you prepare with the Core + CLI. It is comparable to the Go `host` package. -**Experimental:** the protocol is Core's pinned `jacquard-host-v0`, and Core is -installed separately. Each invocation runs one fresh, serial worker. The host -and operating system are trusted. See [limits](../docs/LIMITS.md). +Linux is exercised. macOS should work but is not exercised in CI, and Windows +is not claimed. **Experimental:** the protocol is Core's pinned +`jacquard-host-v0`, and Core is installed separately. Each invocation runs one +fresh, serial worker. The host and operating system are trusted. See +[limits](../docs/LIMITS.md). -## Use +## Install The crate is not published to crates.io. Depend on it from a clone or by Git: @@ -25,6 +29,255 @@ The crate is not published to crates.io. Depend on it from a clone or by Git: jacquard-host = { git = "https://github.com/jbwinters/jacquard-host" } ``` +From a clone, install the pinned Core release (see [setup](../docs/SETUP.md)), +then run the quickstart: + +```sh +eval "$(python3 scripts/core_pin.py install)" +cd rust +cargo run --example jacquard-examples -- quickstart +``` + +Expected output: + +```text +Order total 35: shipping 5 +Order total 75: shipping 0 +``` + +The same command runs the other example projects; see +[run the projects](#run-the-projects). + +## Call a checked function + +```rust,no_run +use jacquard_host::sdk::{PrepareOptions, Runtime, RuntimeOptions}; + +fn shipping() -> Result<(), jacquard_host::sdk::Error> { + let runtime = Runtime::new(RuntimeOptions::default())?; + let program = runtime.prepare(&["rules.jac"], &PrepareOptions::default())?; + let shipping = program.function("shipping-cost")?; + let cost: i64 = shipping.call((35,))?; + + let invocation = shipping.invoke((75,))?; + println!("{}", invocation.render()); + invocation.save(".scratch/shipping.json").expect("saved"); + Ok(()) +} +``` + +The shipping rule is the [quickstart source](../examples/quickstart/rules.jac). +`prepare` takes `.jac` paths (or `.jqd` paths) in dependency order. It uses +Core's public `check`, `hash`, and `run` commands, and keeps a verified cache +keyed by: +- the source bytes; +- the Core executable bytes; +- every prelude file; +- the protocol and preparation format. + +The cache defaults to `.scratch/jacquard-host-rust` under the working +directory; `PrepareOptions::cache_dir` changes it. A changed or incomplete +cache entry is refused rather than repaired. Clones of one `Runtime` share its +version check and prelude discovery, including across threads. Each call runs +one fresh worker. Prepare only first-party declaration sources: preparation +runs Core's `run` command, without service bindings, to populate the store. + +`RuntimeOptions` defaults to `$JACQUARD_BIN` and `$JACQUARD_PRELUDE`. The +runtime refuses a Core whose `--version` differs from the pinned release. Each +Core command must finish within 30 seconds. It runs in its own process group, +which is stopped when the command times out, and on Linux also when it exits. +If its output stays open for a second after it exits (a descendant that left +the group holds it), preparation fails, and the host stops reading and closes +the pipes; the descendant keeps running. + +Arguments are a tuple of up to six values, such as `(35,)` or +`(&policy, &rows)`, or a slice of `&dyn Encode` for more. `call` decodes the result into the type you ask for; `invoke` +keeps the whole `Invocation`. The `_with` variants take a `Cancel`. +`Invocation::save` writes the frames and evidence, including argument and +result values; a new file is created owner-only on Unix, and an existing +file keeps its permissions. + +## Data + +Values map by their Rust type. Before any worker starts, the encoded +arguments are checked against the function's checked signature; Core always +checks the actual wire types too. + +| Rust | Jacquard | +|---|---| +| `i8` to `i64`, `u8` to `u64`, `isize`, `usize` | `Int`; `i64` covers Core's full 63-bit range; out-of-range values are refused | +| `f64` (`f32` to encode) | `Real` as binary64 bits, including NaN payloads and signed zero; an integer at a `Real` parameter is refused | +| `String` (`&str` to encode), `bool`, `protocol::Hash` | `Text`, `Bool`, `Hash` | +| `Vec` (`&[T]` to encode) | `List T` | +| `Option` | `Option T`; `Option>` keeps nested options distinct | +| `()` and tuples up to six items | unit and tuples | +| a struct with `record!` | one constructor, its listed fields in order | +| an enum with `variants!` | the constructors of one data type: unit variants are nullary constructors; tuple and struct variants list their fields in order | +| `sdk::Dynamic` | any of the above by public name, for code with no application type | +| `protocol::Value` | passed through unchanged | + +```rust +use jacquard_host::{record, variants}; + +struct Incident { + id: String, + service: String, + severity: i64, + age: i64, + visible: bool, + owner: Option, +} +record!(Incident = "Incident" { id, service, severity, age, visible, owner }); + +enum Urgency { + Page, + Today, + Watch, +} +variants!(Urgency { Page = "PageNow", Today = "WorkToday", Watch = "WatchOnly" }); + +enum Triage { + Routed { id: String, team: String, urgency: Urgency, reasons: Vec }, + InvalidIncident(String), +} +variants!(Triage { + Routed { id, team, urgency, reasons } = "Routed", + InvalidIncident(id) = "InvalidIncident", +}); +``` + +The listed fields must follow the Jacquard constructor's positional order; +the pinned CLI exports no field types, so the order is not checked against +Core. Constructor names may use display spelling (`WorkToday`) or store +spelling (`work-today`). A constructor that is not in the prepared program is +refused when a value is encoded, and a result built by a different +constructor fails to decode. The macros expand to ordinary `Encode` and +`Decode` implementations, which you may also write by hand with `Encoder` and +`Decoder`. + +Text, collection sizes, and nesting depth are bounded by the selected limits +before a worker starts. `Program::data()` gives each public name's exact +identity. `Data::make` builds a `Dynamic::Variant` after checking that its +constructor exists, and `Invocation::value` decodes any result into `Dynamic`. + +## Services and failures + +```rust +use jacquard_host::host::{HandlerContext, HandlerError}; +use jacquard_host::sdk::{Error, Operations, Program}; + +fn summarize(program: &Program) -> Result<(), Error> { + let documents = Operations::new().on("read-document", |context: &HandlerContext, key: String| { + if context.is_cancelled() { + return Err(HandlerError::completion_unknown("cancelled")); + } + match key.as_str() { + "example" => Ok("A synthetic document supplied by Rust.".to_owned()), + _ => Err(HandlerError::refused("unknown document")), + } + }); + let summarize = program.function_with("summarize", [("Documents", documents)])?; + Ok(()) +} +``` + +An `Operations` value maps operation names (store spelling such as +`read-document`, or Rust spelling such as `read_document`) to closures. One +`Operations` may serve several effects. + +- **Parameters:** a closure may take a leading `&HandlerContext`. The + remaining parameters, up to six, decode from the request into their Rust + types. +- **Results:** a closure returns `Result`, where `T` encodes + the result (`()` for unit). +- **Failures:** return `HandlerError::unsupported`, `refused`, + `outside_failed`, or `completion_unknown` to report that failure. Any other + error raised with `?`, or a panic, is reported to Core as completion unknown + without its message. +- **Binding:** every operation in each checked root effect needs a handler + before the first invocation, including operations a given path never + reaches. A name that is no operation of the program is refused. + `Program::provide` shares services across functions; each function receives + only its own checked effects. + +Closures run on their own threads, so they are `Send + Sync + 'static`; share +state through `Arc` and a lock, as the fulfillment example's warehouse does. +`RuntimeOptions::deadline` (default 30 s) bounds each worker lifetime; see +[handlers and cancellation](#handlers-and-cancellation). + +## Errors + +All SDK failures are an `sdk::Error`: +- `InvalidArgument` is a local refusal, such as a value of the wrong type or + one too large or too deep for a frame. No worker starts. `invoke` returns + only this error; otherwise the `Invocation` holds the raw result. +- `Invocation` wraps an `InvocationError` from `call`, `Invocation::decode`, + and `map_batches`. `Error::failure` classifies it as `DeadlineExceeded`, + `WorkerLost`, `OperationFailed`, or `Other`. +- `Binding` reports an unknown name, a missing handler, or a result that does + not decode into the requested type. In the decode case, `result` keeps the + successful Core result. +- `Preparation` covers checking and the cache. + +`map_batches` runs a pure function whose final parameter and result are Lists +over explicit chunks. It validates every batch before the first runs and +retries nothing. Its `BatchError` names the failed batch and keeps the decoded +values of the batches that completed. + +A request argument that does not decode into a closure's parameter type, or a +wrong argument count, is reported as an outside failure, and that closure does +not run. A result that cannot be encoded is also reported as an outside +failure, but only after the closure returned, so any action it took has +happened. Neither says anything about earlier operations in the invocation. + +## Run the projects + +```sh +cd rust +cargo run --example jacquard-examples -- planner +cargo run --example jacquard-examples -- triage +cargo run --example jacquard-examples -- fulfillment +cargo run --example jacquard-examples -- fulfillment +``` + +- Planner returns score 16. +- Triage returns PAGE, TODAY, and WATCH. +- Fulfillment returns Fulfilled, then DuplicateOrder with the same receipt. + +The command reads the rules and synthetic inputs from the repository's +[`examples/`](../examples/README.md), so it runs from a clone. Custom input +JSON may follow the project name. `--inspect PATH` saves frames; triage adds a +numbered suffix per batch. State defaults to `.scratch/rust-examples/`. + +Try explicit recovery in a fresh work directory: + +```sh +cargo run --example jacquard-examples -- --workdir .scratch/rust-recovery --fault publish-failed fulfillment +cargo run --example jacquard-examples -- --workdir .scratch/rust-recovery --deliver-pending fulfillment +cargo run --example jacquard-examples -- --workdir .scratch/rust-recovery --deliver-pending fulfillment +``` + +The first command exits 1 with E1613 and one pending booking. Reconciliation +then delivers one receipt, and the next run delivers zero. The `after-commit` +and `after-delivery` faults exercise E1612 around the snapshot commits. + +The example's `Warehouse` is a single-process service. It writes bookings and +outbox entries together as one atomic JSON snapshot, and a simulated delivery +sink deduplicates receipts. It has no cross-process locking and makes no +power-loss durability claim. + +## Modules + +| Module | Contents | +|---|---| +| `sdk` | The SDK: `Runtime`, `Program`, `Data`, `Function`, `Invocation`, `Operations`, `Dynamic`, and the `Encode` and `Decode` traits, with the `record!` and `variants!` macros at the crate root | +| `host` | Low-level `Host`, `OperationRegistry`, `HandlerError`, and `InvocationResult` for exact identities | +| `carrier` | The bounded process carrier for `jacquard host worker`, and `Cancel` | +| `protocol`, `json` | Strict JSON, v0 values and types, limits, the `Session` state machine | +| `compat`, `kit` | The generated pin constants, and the embedded pinned conformance kit and its replayer | + +## The protocol-level host + The example below invokes a checked function whose single operation is served by a Rust handler: @@ -63,7 +316,7 @@ status, bounded stderr, and host evidence (`jacquard-host-rust-evidence-v0`). ## Handlers and cancellation -A handler returns a value or a `HandlerError`. The four structured categories +These rules apply to SDK closures and to low-level handlers. A handler returns a value or a `HandlerError`. The four structured categories are `unsupported`, `refused`, `outside_failed`, and `completion_unknown`. `HandlerError::Unknown` and a panic are both reported to Core as completion unknown, and their message is never sent. Panics are caught only with the @@ -96,7 +349,10 @@ cargo test cargo test -- --nocapture ``` -The live tests in `tests/live.rs` skip without `JACQUARD_BIN` and +`tests/sdk_examples.rs` also runs the built example binary, so run it as part +of a plain `cargo test`, which builds the examples, rather than alone with +`--test`. The live tests in `tests/live.rs`, `tests/sdk_live.rs`, and +`tests/sdk_examples.rs` skip without `JACQUARD_BIN` and `JACQUARD_PRELUDE` (set `JACQUARD_REQUIRE_LIVE` to make them fail instead). The offline tests drive `jacquard-host-standin`, a Unix-only test fixture binary that replays recordings or simulates faulty workers. It is not part of diff --git a/rust/examples/jacquard-examples.rs b/rust/examples/jacquard-examples.rs new file mode 100644 index 0000000..d217f0f --- /dev/null +++ b/rust/examples/jacquard-examples.rs @@ -0,0 +1,715 @@ +//! Runs the repository's example Jacquard projects through the Rust SDK: +//! quickstart, planner, triage, and fulfillment. The same rules run through +//! the Python, TypeScript, and Go SDKs. +//! +//! ```text +//! cargo run --example jacquard-examples -- [--workdir DIR] [--inspect PATH] [--fault NAME] [--deliver-pending] PROJECT [INPUT.json] +//! ``` +//! +//! Core is found through `JACQUARD_BIN` and `JACQUARD_PRELUDE`. The exit +//! status is 1 for a Jacquard or SDK failure and 2 for usage or other errors. + +use std::io::Write; +use std::path::{Path, PathBuf}; +use std::sync::{Arc, Mutex, MutexGuard}; + +use jacquard_host::host::HandlerError; +use jacquard_host::json::{Json, Object, decode, stringify}; +use jacquard_host::sdk::{BatchError, Error, Function, Invocation, Operations, PrepareOptions, Program, Runtime, RuntimeOptions}; +use jacquard_host::{record, variants}; + +/// The example projects in presentation order. +pub const PROJECTS: [&str; 4] = ["quickstart", "planner", "triage", "fulfillment"]; + +/// The repository's example projects. +fn examples() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")).join("../examples") +} + +/// A failure, classified for the exit status. +#[derive(Debug)] +pub enum Failure { + Sdk(Error), + Other(String), +} + +impl std::fmt::Display for Failure { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Failure::Sdk(error) => error.fmt(f), + Failure::Other(message) => f.write_str(message), + } + } +} + +impl From for Failure { + fn from(error: Error) -> Self { + Failure::Sdk(error) + } +} + +impl From> for Failure { + fn from(error: BatchError) -> Self { + Failure::Sdk(error.error) + } +} + +impl From for Failure { + fn from(error: std::io::Error) -> Self { + Failure::Other(error.to_string()) + } +} + +fn other(message: impl Into) -> Failure { + Failure::Other(message.into()) +} + +// JSON input and output. + +fn field<'a>(object: &'a Object, key: &str) -> Result<&'a Json, Failure> { + object.get(key).ok_or_else(|| other(format!("input: missing {key}"))) +} + +fn text(object: &Object, key: &str) -> Result { + field(object, key)?.as_str().map(str::to_owned).ok_or_else(|| other(format!("input: {key} must be text"))) +} + +fn int(object: &Object, key: &str) -> Result { + field(object, key)?.as_int().ok_or_else(|| other(format!("input: {key} must be an integer"))) +} + +fn boolean(object: &Object, key: &str) -> Result { + match field(object, key)? { + Json::Bool(value) => Ok(*value), + _ => Err(other(format!("input: {key} must be true or false"))), + } +} + +fn objects<'a>(object: &'a Object, key: &str) -> Result, Failure> { + let items = field(object, key)?.as_array().ok_or_else(|| other(format!("input: {key} must be a list")))?; + items.iter().map(|item| item.as_object().ok_or_else(|| other(format!("input: {key} must hold objects")))).collect() +} + +fn tagged(tag: &str, fields: impl IntoIterator) -> Json { + let mut object: Object = fields.into_iter().map(|(key, value)| (key.to_owned(), value)).collect(); + object.insert("$tag".into(), tag.into()); + Json::Object(object) +} + +fn pretty(value: &Json, indent: usize, out: &mut String) { + let pad = |depth: usize| " ".repeat(depth); + match value { + Json::Array(items) if !items.is_empty() => { + out.push_str("[\n"); + for (index, item) in items.iter().enumerate() { + out.push_str(&pad(indent + 1)); + pretty(item, indent + 1, out); + out.push_str(if index + 1 < items.len() { ",\n" } else { "\n" }); + } + out.push_str(&pad(indent)); + out.push(']'); + } + Json::Object(object) if !object.is_empty() => { + out.push_str("{\n"); + for (index, (key, item)) in object.iter().enumerate() { + out.push_str(&pad(indent + 1)); + out.push_str(&stringify(&Json::Str(key.clone()), 2).expect("a key")); + out.push_str(": "); + pretty(item, indent + 1, out); + out.push_str(if index + 1 < object.len() { ",\n" } else { "\n" }); + } + out.push_str(&pad(indent)); + out.push('}'); + } + _ => out.push_str(&stringify(value, 2).expect("a scalar")), + } +} + +fn print_json(out: &mut impl Write, value: &Json) -> Result<(), Failure> { + let mut text = String::new(); + pretty(value, 0, &mut text); + writeln!(out, "{text}")?; + Ok(()) +} + +// Planner. + +pub struct Job { + pub id: String, + pub effort: i64, + pub value: i64, + pub latest: i64, +} +record!(Job = "Job" { id, effort, value, latest }); + +pub struct Assignment { + pub id: String, + pub slot: i64, +} +record!(Assignment = "Assignment" { id, slot }); + +pub struct Plan { + pub score: i64, + pub assignments: Vec, +} +record!(Plan = "Plan" { score, assignments }); + +pub enum Planning { + Planned { plan: Plan }, + InvalidPlan { reason: String }, +} +variants!(Planning { Planned { plan } = "Planned", InvalidPlan { reason } = "InvalidPlan" }); + +impl Planning { + fn to_json(&self) -> Json { + match self { + Planning::Planned { plan } => { + let assignments = plan + .assignments + .iter() + .map(|a| Json::Object(Object::from([("id".into(), a.id.as_str().into()), ("slot".into(), a.slot.into())]))) + .collect(); + let plan = Object::from([("assignments".into(), Json::Array(assignments)), ("score".into(), plan.score.into())]); + tagged("Planned", [("plan", Json::Object(plan))]) + } + Planning::InvalidPlan { reason } => tagged("InvalidPlan", [("reason", reason.as_str().into())]), + } + } +} + +// Triage. + +pub struct TriagePolicy { + pub page: i64, + pub stale: i64, +} +record!(TriagePolicy = "TriagePolicy" { page, stale }); + +pub struct Incident { + pub id: String, + pub service: String, + pub severity: i64, + pub age: i64, + pub visible: bool, + pub owner: Option, +} +record!(Incident = "Incident" { id, service, severity, age, visible, owner }); + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Urgency { + Page, + Today, + Watch, +} +variants!(Urgency { Page = "PageNow", Today = "WorkToday", Watch = "WatchOnly" }); + +pub enum Triage { + Routed { id: String, team: String, urgency: Urgency, reasons: Vec }, + InvalidIncident { id: String }, +} +variants!(Triage { Routed { id, team, urgency, reasons } = "Routed", InvalidIncident { id } = "InvalidIncident" }); + +impl Triage { + fn to_json(&self) -> Json { + match self { + Triage::Routed { id, team, urgency, reasons } => { + let urgency = match urgency { + Urgency::Page => "PAGE", + Urgency::Today => "TODAY", + Urgency::Watch => "WATCH", + }; + tagged( + "Routed", + [ + ("id", id.as_str().into()), + ("team", team.as_str().into()), + ("urgency", urgency.into()), + ("reasons", Json::Array(reasons.iter().map(|reason| reason.as_str().into()).collect())), + ], + ) + } + Triage::InvalidIncident { id } => tagged("InvalidIncident", [("id", id.as_str().into())]), + } + } +} + +// Fulfillment. + +#[derive(Clone)] +pub struct Line { + pub sku: String, + pub units: i64, +} +record!(Line = "Line" { sku, units }); + +#[derive(Clone)] +pub struct Order { + pub id: String, + pub lines: Vec, +} +record!(Order = "Order" { id, lines }); + +pub enum Stock { + Stock(i64), + UnknownSku, +} +variants!(Stock { Stock(available) = "Stock", UnknownSku = "UnknownSku" }); + +pub enum Reservation { + BookingPending, + Booked(String), + RepeatBooking(String), + BookingConflict, + StockChanged, +} +variants!(Reservation { + BookingPending = "BookingPending", + Booked(receipt) = "Booked", + RepeatBooking(receipt) = "RepeatBooking", + BookingConflict = "BookingConflict", + StockChanged = "StockChanged", +}); + +pub enum Fulfillment { + Fulfilled { receipt: String }, + DuplicateOrder { receipt: String }, + OrderRejected { reason: String }, +} +variants!(Fulfillment { + Fulfilled { receipt } = "Fulfilled", + DuplicateOrder { receipt } = "DuplicateOrder", + OrderRejected { reason } = "OrderRejected", +}); + +impl Fulfillment { + fn to_json(&self) -> Json { + match self { + Fulfillment::Fulfilled { receipt } => tagged("Fulfilled", [("receipt", receipt.as_str().into())]), + Fulfillment::DuplicateOrder { receipt } => tagged("DuplicateOrder", [("receipt", receipt.as_str().into())]), + Fulfillment::OrderRejected { reason } => tagged("OrderRejected", [("reason", reason.as_str().into())]), + } + } +} + +const INVENTORY_SCHEMA: &str = "jacquard-example-inventory-v1"; + +#[derive(Clone)] +struct Booking { + id: String, + payload: String, + receipt: String, +} + +#[derive(Clone)] +struct Outbox { + id: String, + receipt: String, + delivered: bool, +} + +#[derive(Clone)] +struct Inventory { + stock: Vec<(String, i64)>, + bookings: Vec, + outbox: Vec, + deliveries: Vec, +} + +impl Inventory { + fn to_json(&self) -> Json { + let object = |pairs: Vec<(&str, Json)>| Json::Object(pairs.into_iter().map(|(key, value)| (key.to_owned(), value)).collect()); + object(vec![ + ("schema", INVENTORY_SCHEMA.into()), + ( + "stock", + Json::Array( + self.stock + .iter() + .map(|(sku, available)| object(vec![("sku", sku.as_str().into()), ("available", (*available).into())])) + .collect(), + ), + ), + ( + "bookings", + Json::Array( + self.bookings + .iter() + .map(|b| { + object(vec![ + ("id", b.id.as_str().into()), + ("payload", b.payload.as_str().into()), + ("receipt", b.receipt.as_str().into()), + ]) + }) + .collect(), + ), + ), + ( + "outbox", + Json::Array( + self.outbox + .iter() + .map(|o| { + object(vec![ + ("id", o.id.as_str().into()), + ("receipt", o.receipt.as_str().into()), + ("delivered", Json::Bool(o.delivered)), + ]) + }) + .collect(), + ), + ), + ("deliveries", Json::Array(self.deliveries.iter().map(|d| d.as_str().into()).collect())), + ]) + } + + fn from_json(value: &Json) -> Option { + let object = value.as_object()?; + if object.get("schema")?.as_str()? != INVENTORY_SCHEMA { + return None; + } + let list = |key: &str| object.get(key)?.as_array(); + let get = |item: &Json, key: &str| -> Option { Some(item.as_object()?.get(key)?.as_str()?.to_owned()) }; + Some(Inventory { + stock: list("stock")? + .iter() + .map(|item| Some((get(item, "sku")?, item.as_object()?.get("available")?.as_int()?))) + .collect::>()?, + bookings: list("bookings")? + .iter() + .map(|item| Some(Booking { id: get(item, "id")?, payload: get(item, "payload")?, receipt: get(item, "receipt")? })) + .collect::>()?, + outbox: list("outbox")? + .iter() + .map(|item| { + let Json::Bool(delivered) = *item.as_object()?.get("delivered")? else { return None }; + Some(Outbox { id: get(item, "id")?, receipt: get(item, "receipt")?, delivered }) + }) + .collect::>()?, + deliveries: list("deliveries")?.iter().map(|item| item.as_str().map(str::to_owned)).collect::>()?, + }) + } +} + +/// A single-process synthetic Inventory and Dispatch service. Bookings and +/// outbox entries are committed together as one atomic JSON snapshot; +/// deliveries are deduplicated by receipt. It has no cross-process locking +/// or power-loss durability claim. +pub struct Warehouse { + /// Injects `publish-failed`, `after-commit`, or `after-delivery`. + pub fault: String, + path: PathBuf, + state: Inventory, +} + +/// The canonical payload of an order: its lines sorted by SKU. +fn payload(order: &Order) -> String { + let mut lines = order.lines.clone(); + lines.sort_by(|a, b| a.sku.cmp(&b.sku)); + let pairs = lines.iter().map(|line| Json::Array(vec![line.sku.as_str().into(), line.units.into()])).collect(); + stringify(&Json::Array(pairs), 4).expect("a flat payload") +} + +impl Warehouse { + /// Loads the snapshot at `path` or starts synthetic stock. + pub fn open(path: PathBuf) -> Result { + let state = match std::fs::read(&path) { + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Inventory { + stock: vec![("sensor".into(), 20), ("cable".into(), 30), ("battery".into(), 15)], + bookings: Vec::new(), + outbox: Vec::new(), + deliveries: Vec::new(), + }, + Err(error) => return Err(error.into()), + Ok(raw) => { + decode(&raw, 16).ok().as_ref().and_then(Inventory::from_json).ok_or_else(|| other("inventory snapshot version mismatch"))? + } + }; + Ok(Warehouse { fault: String::new(), path, state }) + } + + fn commit(&mut self, next: Inventory) -> std::io::Result<()> { + if let Some(parent) = self.path.parent() { + std::fs::create_dir_all(parent)?; + } + let text = stringify(&next.to_json(), 8).map_err(std::io::Error::other)?; + // A fresh owner-only file, created exclusively, then renamed over the + // snapshot. + static NEXT: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); + let nanos = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |elapsed| elapsed.subsec_nanos()); + let suffix = format!("{}-{nanos:08x}-{}", std::process::id(), NEXT.fetch_add(1, std::sync::atomic::Ordering::SeqCst)); + let temporary = self.path.with_extension(format!("{suffix}.tmp")); + let mut options = std::fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + std::os::unix::fs::OpenOptionsExt::mode(&mut options, 0o600); + let written = options.open(&temporary).and_then(|mut file| file.write_all(format!("{text}\n").as_bytes())); + if let Err(error) = written { + let _ = std::fs::remove_file(&temporary); + return Err(error); + } + let renamed = std::fs::rename(&temporary, &self.path); + let _ = std::fs::remove_file(&temporary); + renamed?; + self.state = next; + Ok(()) + } + + fn find(&self, id: &str) -> Option<&Booking> { + self.state.bookings.iter().find(|booking| booking.id == id) + } + + /// Serves `Inventory.booking-status`. + pub fn booking_status(&self, order: &Order) -> Reservation { + match self.find(&order.id) { + None => Reservation::BookingPending, + Some(previous) if previous.payload == payload(order) => Reservation::RepeatBooking(previous.receipt.clone()), + Some(_) => Reservation::BookingConflict, + } + } + + /// Serves `Inventory.lookup-stock`. + pub fn lookup_stock(&self, sku: &str) -> Stock { + self.state.stock.iter().find(|(name, _)| name == sku).map_or(Stock::UnknownSku, |(_, available)| Stock::Stock(*available)) + } + + /// Serves `Inventory.reserve-order`, committing booking and outbox together. + pub fn reserve_order(&mut self, order: &Order) -> Result { + let mut seen = std::collections::BTreeSet::new(); + let valid = !order.id.is_empty() + && (1..=20).contains(&order.lines.len()) + && order.lines.iter().all(|line| seen.insert(line.sku.clone()) && (1..=20).contains(&line.units)); + if !valid { + return Err(HandlerError::refused("invalid synthetic reservation")); + } + if self.find(&order.id).is_some() { + return Ok(self.booking_status(order)); + } + let mut next = self.state.clone(); + for line in &order.lines { + match next.stock.iter_mut().find(|(sku, _)| *sku == line.sku) { + Some((_, available)) if *available >= line.units => *available -= line.units, + _ => return Ok(Reservation::StockChanged), + } + } + let body = payload(order); + let receipt = jacquard_host::sha256::hex_digest(format!("{}\0{body}", order.id).as_bytes()); + next.bookings.push(Booking { id: order.id.clone(), payload: body, receipt: receipt.clone() }); + next.outbox.push(Outbox { id: order.id.clone(), receipt: receipt.clone(), delivered: false }); + self.commit(next)?; + if self.fault == "after-commit" { + return Err(HandlerError::Unknown); + } + Ok(Reservation::Booked(receipt)) + } + + /// Serves `Dispatch.publish-booking`. + pub fn publish_booking(&mut self, id: &str, receipt: &str) -> Result<(), HandlerError> { + if self.fault == "publish-failed" { + return Err(HandlerError::outside_failed("synthetic delivery unavailable; outbox retained")); + } + let mut next = self.state.clone(); + let Some(entry) = next.outbox.iter_mut().find(|entry| entry.receipt == receipt && entry.id == id) else { + return Err(HandlerError::refused("no matching outbox entry")); + }; + entry.delivered = true; + if !next.deliveries.iter().any(|delivery| delivery == receipt) { + next.deliveries.push(receipt.to_owned()); + } + self.commit(next)?; + if self.fault == "after-delivery" { + return Err(HandlerError::Unknown); + } + Ok(()) + } + + /// Publishes every undelivered outbox entry explicitly, returning how + /// many were delivered, and the error that stopped it. + pub fn deliver_pending(&mut self) -> (usize, Result<(), HandlerError>) { + let pending: Vec = self.state.outbox.iter().filter(|entry| !entry.delivered).cloned().collect(); + for (delivered, entry) in pending.iter().enumerate() { + if let Err(error) = self.publish_booking(&entry.id, &entry.receipt) { + return (delivered, Err(error)); + } + } + (pending.len(), Ok(())) + } + + /// Counts bookings, pending deliveries, and deliveries. + pub fn summary(&self) -> Json { + let pending = self.state.outbox.iter().filter(|entry| !entry.delivered).count(); + Json::Object(Object::from([ + ("bookings".into(), Json::Int(self.state.bookings.len() as i64)), + ("deliveries".into(), Json::Int(self.state.deliveries.len() as i64)), + ("pendingDeliveries".into(), Json::Int(pending as i64)), + ])) + } +} + +fn describe(error: &HandlerError) -> String { + match error { + HandlerError::Failure(category, message) => format!("{}: {message}", category.as_str()), + HandlerError::Unknown => "completion unknown".into(), + } +} + +fn locked(warehouse: &Mutex) -> MutexGuard<'_, Warehouse> { + warehouse.lock().unwrap_or_else(|poisoned| poisoned.into_inner()) +} + +/// Binds `fulfillment.place` to a shared warehouse serving both effects. +pub fn fulfillment(program: &Program, warehouse: &Arc>) -> Result { + let (a, b, c, d) = (warehouse.clone(), warehouse.clone(), warehouse.clone(), warehouse.clone()); + let service = Operations::new() + .on("booking-status", move |order: Order| Ok(locked(&a).booking_status(&order))) + .on("lookup-stock", move |sku: String| Ok(locked(&b).lookup_stock(&sku))) + .on("reserve-order", move |order: Order| locked(&c).reserve_order(&order)) + .on("publish-booking", move |id: String, receipt: String| locked(&d).publish_booking(&id, &receipt)); + program.function_with("fulfillment.place", [("Inventory", service.clone()), ("Dispatch", service)]) +} + +/// Runs one command line, writing its output to `out`. +pub fn run(arguments: &[String], out: &mut impl Write) -> Result<(), Failure> { + let mut workdir = PathBuf::from(".scratch/rust-examples"); + let (mut inspect, mut fault, mut deliver_pending) = (None, String::new(), false); + let mut positional = Vec::new(); + let mut arguments = arguments.iter(); + while let Some(argument) = arguments.next() { + let mut value = || arguments.next().cloned().ok_or_else(|| other(format!("{argument} needs a value"))); + match argument.as_str() { + "--workdir" => workdir = PathBuf::from(value()?), + "--inspect" => inspect = Some(value()?), + "--fault" => fault = value()?, + "--deliver-pending" => deliver_pending = true, + flag if flag.starts_with("--") => return Err(other(format!("unknown flag {flag}"))), + _ => positional.push(argument.clone()), + } + } + let project = positional.first().map(String::as_str).unwrap_or_default(); + if !PROJECTS.contains(&project) || positional.len() > 2 { + return Err(other("choose quickstart, triage, planner, or fulfillment")); + } + if !["", "publish-failed", "after-commit", "after-delivery"].contains(&fault.as_str()) { + return Err(other("unknown fault")); + } + let root = std::path::absolute(&workdir)?; + let rules = examples().join(project).join("rules.jac"); + let runtime = Runtime::new(RuntimeOptions::default())?; + let program = runtime.prepare(&[&rules], &PrepareOptions { cache_dir: Some(root.join("programs")) })?; + if project == "quickstart" { + let shipping = program.function("shipping-cost")?; + for total in [35, 75] { + let cost: i64 = shipping.call((total,))?; + writeln!(out, "Order total {total}: shipping {cost}")?; + } + return Ok(()); + } + let input = positional.get(1).map_or_else(|| examples().join(project).join("input.json"), PathBuf::from); + let raw = std::fs::read(&input)?; + let input = decode(&raw, 16).map_err(|error| other(format!("input: {error}")))?; + let input = input.as_object().ok_or_else(|| other("input must be a JSON object"))?; + let save = |invocation: &Invocation, path: Option| -> Result<(), Failure> { + if let Some(path) = path { + invocation.save(path)?; + } + Ok(()) + }; + match project { + "planner" => { + let jobs = objects(input, "jobs")? + .into_iter() + .map(|job| { + Ok(Job { id: text(job, "id")?, effort: int(job, "effort")?, value: int(job, "value")?, latest: int(job, "latest")? }) + }) + .collect::, Failure>>()?; + let capacities = field(input, "capacities")?.as_array().ok_or_else(|| other("input: capacities must be a list"))?; + let capacities = capacities + .iter() + .map(|cap| cap.as_int().ok_or_else(|| other("input: capacities must be integers"))) + .collect::, _>>()?; + let invocation = program.function("planner.plan")?.invoke((&jobs, &capacities))?; + save(&invocation, inspect)?; + print_json(out, &invocation.decode::()?.to_json())?; + writeln!(out, "{}", invocation.render())?; + } + "triage" => { + let policy = field(input, "policy")?.as_object().ok_or_else(|| other("input: policy must be an object"))?; + let policy = TriagePolicy { page: int(policy, "page")?, stale: int(policy, "stale")? }; + let incidents = objects(input, "incidents")? + .into_iter() + .map(|incident| { + // A missing owner is null, as in the Go command. + let owner = match incident.get("owner").unwrap_or(&Json::Null) { + Json::Null => None, + owner => Some(owner.as_str().ok_or_else(|| other("input: owner must be text or null"))?.to_owned()), + }; + Ok(Incident { + id: text(incident, "id")?, + service: text(incident, "service")?, + severity: int(incident, "severity")?, + age: int(incident, "age")?, + visible: boolean(incident, "visible")?, + owner, + }) + }) + .collect::, Failure>>()?; + let route = program.function("triage.batch")?; + let results: Vec = match inspect { + None => route.map_batches((&policy,), &incidents, 8)?, + Some(inspect) => { + let mut results = Vec::new(); + for (index, chunk) in incidents.chunks(8).enumerate() { + let invocation = route.invoke((&policy, chunk))?; + save(&invocation, Some(format!("{inspect}.{index}.json")))?; + results.extend(invocation.decode::>()?); + } + results + } + }; + print_json(out, &Json::Array(results.iter().map(Triage::to_json).collect()))?; + } + _ => { + let lines = objects(input, "lines")? + .into_iter() + .map(|line| Ok(Line { sku: text(line, "sku")?, units: int(line, "units")? })) + .collect::, Failure>>()?; + let order = Order { id: text(input, "id")?, lines }; + let mut warehouse = Warehouse::open(root.join("inventory.json"))?; + warehouse.fault = fault; + let warehouse = Arc::new(Mutex::new(warehouse)); + let outcome = (|| { + if deliver_pending { + // Like the Go command, a failed delivery prints no count. + let (delivered, result) = locked(&warehouse).deliver_pending(); + result.map_err(|error| other(format!("delivery failed: {}", describe(&error))))?; + let count = Json::Object(Object::from([("delivered".into(), Json::Int(delivered as i64))])); + writeln!(out, "{}", stringify(&count, 2).expect("flat"))?; + return Ok(()); + } + let invocation = fulfillment(&program, &warehouse)?.invoke((&order,))?; + save(&invocation, inspect)?; + let result = invocation.decode::(); + if let Ok(result) = &result { + print_json(out, &result.to_json())?; + } + writeln!(out, "{}", invocation.render())?; + result.map(drop).map_err(Failure::from) + })(); + writeln!(out, "{}", stringify(&locked(&warehouse).summary(), 2).expect("flat"))?; + outcome?; + } + } + Ok(()) +} + +#[allow(dead_code)] +fn main() { + let arguments: Vec = std::env::args().skip(1).collect(); + if let Err(failure) = run(&arguments, &mut std::io::stdout()) { + eprintln!("{failure}"); + std::process::exit(match failure { + Failure::Sdk(_) => 1, + Failure::Other(_) => 2, + }); + } +} diff --git a/rust/src/carrier.rs b/rust/src/carrier.rs index 0c39467..30041a9 100644 --- a/rust/src/carrier.rs +++ b/rust/src/carrier.rs @@ -576,7 +576,7 @@ fn read_frame(stdout: &mut impl Read, limits: &Limits) -> FrameResult { /// readiness with `poll` in short slices and gives up once the worker is /// disposed or this pipe is stopped, so a descendant holding the other end /// cannot pin the thread or keep receiving data. -struct Pipe { +pub(crate) struct Pipe { inner: T, #[cfg_attr(not(unix), allow(dead_code))] stops: [Arc; 2], @@ -584,7 +584,7 @@ struct Pipe { } impl Pipe { - fn new(inner: T, all: &Arc, own: &Arc) -> Pipe { + pub(crate) fn new(inner: T, all: &Arc, own: &Arc) -> Pipe { Pipe { inner, stops: [Arc::clone(all), Arc::clone(own)], gate: None } } @@ -716,23 +716,23 @@ mod sys { } #[cfg(unix)] -fn isolate(command: &mut Command) { +pub(crate) fn isolate(command: &mut Command) { use std::os::unix::process::CommandExt; command.process_group(0); } #[cfg(not(unix))] -fn isolate(_: &mut Command) {} +pub(crate) fn isolate(_: &mut Command) {} /// Kills an unreaped worker and its process group. -fn kill_tree(child: &mut Child) { +pub(crate) fn kill_tree(child: &mut Child) { #[cfg(unix)] kill_group(child.id()); let _ = child.kill(); } #[cfg(unix)] -fn kill_group(pid: u32) { +pub(crate) fn kill_group(pid: u32) { // SAFETY: kill(2) with a negative pid signals a process group; the caller // guarantees the leader is unreaped, so the group id cannot be reused. unsafe { @@ -741,7 +741,7 @@ fn kill_group(pid: u32) { } #[cfg(not(unix))] -fn kill_group(_: u32) {} +pub(crate) fn kill_group(_: u32) {} /// Test-only: makes Linux take the exit wait used where `waitid` is /// unavailable. Not part of the adapter's interface. @@ -751,7 +751,7 @@ pub static TEST_WITHOUT_WAITID: AtomicBool = AtomicBool::new(false); /// Whether the worker has exited, without reaping it: `Some(true)` once it /// has, `Some(false)` while it runs, `None` where the platform cannot tell. #[cfg(target_os = "linux")] -fn exited_unreaped(pid: u32) -> Option { +pub(crate) fn exited_unreaped(pid: u32) -> Option { if TEST_WITHOUT_WAITID.load(Ordering::SeqCst) { return None; } @@ -777,7 +777,7 @@ fn exited_unreaped(pid: u32) -> Option { } #[cfg(not(target_os = "linux"))] -fn exited_unreaped(_: u32) -> Option { +pub(crate) fn exited_unreaped(_: u32) -> Option { None } diff --git a/rust/src/host.rs b/rust/src/host.rs index dfaf776..74d058e 100644 --- a/rust/src/host.rs +++ b/rust/src/host.rs @@ -48,6 +48,14 @@ impl HandlerError { } } +/// Any other error, such as an I/O error raised with `?`, is reported to +/// Core as completion unknown, without its message. +impl From for HandlerError { + fn from(_: E) -> Self { + HandlerError::Unknown + } +} + /// What a handler sees: whether the invocation is still live. #[derive(Clone, Debug)] pub struct HandlerContext { @@ -57,6 +65,11 @@ pub struct HandlerContext { } impl HandlerContext { + #[cfg(test)] + pub(crate) fn for_test() -> Self { + HandlerContext { expired: Arc::default(), ended: Arc::default(), cancel: Cancel::new() } + } + /// Whether the worker expired, the caller cancelled, or the invocation /// ended. Handlers should stop; the host no longer waits for them. pub fn is_cancelled(&self) -> bool { diff --git a/rust/src/lib.rs b/rust/src/lib.rs index 073144d..cdc48e4 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -1,8 +1,9 @@ //! An independent Rust adapter for Jacquard Core's experimental //! `jacquard-host-v0` protocol: strict codecs, the host session state //! machine, a bounded process carrier for `jacquard host worker`, a low-level -//! [`host::Host`] with a closed operation registry, and a replayer for Core's -//! pinned conformance kit. Standard library only. +//! [`host::Host`] with a closed operation registry, a replayer for Core's +//! pinned conformance kit, and an [`sdk`] that prepares programs, maps native +//! Rust values, and binds closures to operations. Standard library only. pub mod carrier; pub mod compat; @@ -11,7 +12,10 @@ pub mod host; pub mod json; pub mod kit; pub mod protocol; -mod sha256; +pub mod sdk; +/// SHA-256, for the examples; not part of the adapter's interface. +#[doc(hidden)] +pub mod sha256; pub use error::ProtocolError; diff --git a/rust/src/sdk/core.rs b/rust/src/sdk/core.rs new file mode 100644 index 0000000..1dcc292 --- /dev/null +++ b/rust/src/sdk/core.rs @@ -0,0 +1,319 @@ +//! Bounded runs of Core's public commands, and digests of prepared files. + +use std::collections::BTreeMap; +use std::ffi::OsStr; +use std::io::Read; +use std::path::{Path, PathBuf}; +use std::process::{Command, Stdio}; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::thread; +use std::time::{Duration, Instant}; + +use crate::carrier::{Pipe, exited_unreaped, isolate, kill_group, kill_tree}; +use crate::json::{Json, stringify}; +use crate::sha256::hex_digest; + +use super::Error; + +/// Each Core command must finish within this time. +const COMMAND_TIMEOUT: Duration = Duration::from_secs(30); +const STDOUT_LIMIT: usize = 8 << 20; +const STDERR_LIMIT: usize = 4096; + +/// Reads up to `limit` bytes, then drains and discards the rest. +fn bounded_read(mut source: impl Read, limit: usize) -> (Vec, bool) { + let mut kept = Vec::new(); + let mut overflow = false; + let mut buffer = [0u8; 8192]; + loop { + match source.read(&mut buffer) { + Ok(0) => return (kept, overflow), + Ok(n) => { + let room = limit - kept.len(); + if n > room { + overflow = true; + } + kept.extend_from_slice(&buffer[..n.min(room)]); + } + Err(error) if error.kind() == std::io::ErrorKind::Interrupted => {} + Err(_) => return (kept, overflow), + } + } +} + +/// Runs one Core subcommand with the prelude set, returning its stdout. The +/// command runs in its own process group, which is killed when it times out, +/// and on Linux also when it exits. Output is read through stoppable pipes: +/// once the command has exited and its output stays open for the grace +/// period (a descendant that left the group still holds it), the readers stop +/// and the pipes close. +pub(crate) fn run(binary: &Path, prelude: &Path, arguments: &[&OsStr]) -> Result { + let name = arguments.first().map_or_else(String::new, |argument| argument.to_string_lossy().into_owned()); + let failed = |detail: String| Error::preparation(format!("Core {name} failed: {detail}")); + let mut command = Command::new(binary); + command.args(arguments).env("JACQUARD_PRELUDE", prelude).stdin(Stdio::null()).stdout(Stdio::piped()).stderr(Stdio::piped()); + isolate(&mut command); + let mut running = Running { child: command.spawn().map_err(|error| failed(error.to_string()))?, reaped: false }; + let stop = Arc::new(AtomicBool::new(false)); + // Declared after `running`, so on any early return the readers stop + // first and the command's group is then killed and reaped. + let mut readers = Readers { stop: stop.clone(), stdout: None, stderr: None }; + let stdout = Pipe::new(running.child.stdout.take().expect("piped stdout"), &stop, &stop); + readers.stdout = Some(spawn_reader(stdout, STDOUT_LIMIT).map_err(|error| failed(format!("cannot start a reader: {error}")))?); + let stderr = Pipe::new(running.child.stderr.take().expect("piped stderr"), &stop, &stop); + readers.stderr = Some(spawn_reader(stderr, STDERR_LIMIT).map_err(|error| failed(format!("cannot start a reader: {error}")))?); + let deadline = Instant::now() + COMMAND_TIMEOUT; + let status = loop { + let child = &mut running.child; + match exited_unreaped(child.id()) { + Some(true) => { + // The leader is unreaped, so its group id is still ours. + kill_group(child.id()); + running.reaped = true; + break child.wait(); + } + Some(false) => {} + None => match child.try_wait() { + Ok(Some(status)) => { + running.reaped = true; + break Ok(status); + } + Ok(None) => {} + Err(error) => { + // The state is unknown; never signal a group that may + // no longer be ours. + running.reaped = true; + break Err(error); + } + }, + } + if Instant::now() >= deadline { + drop(readers); + drop(running); + return Err(failed(format!("no exit within {} seconds", COMMAND_TIMEOUT.as_secs()))); + } + thread::sleep(Duration::from_millis(10)); + }; + let status = status.map_err(|error| failed(error.to_string()))?; + let Some(((stdout, overflow), (stderr, _))) = readers.finish(Instant::now() + OUTPUT_GRACE) else { + return Err(failed("its output did not close".into())); + }; + if !status.success() { + let detail = String::from_utf8_lossy(&stderr).trim().to_owned(); + return Err(failed(if detail.is_empty() { status.to_string() } else { detail })); + } + if overflow { + return Err(Error::preparation(format!("Core {name} output exceeds 8 MiB"))); + } + String::from_utf8(stdout).map_err(|_| Error::preparation(format!("Core {name} output is not UTF-8"))) +} + +/// How long output may stay open after the command exits. +const OUTPUT_GRACE: Duration = Duration::from_secs(1); + +type Output = (Vec, bool); + +/// The running command; dropping it before it is reaped kills its process +/// group and reaps it. +struct Running { + child: std::process::Child, + reaped: bool, +} + +impl Drop for Running { + fn drop(&mut self) { + if !self.reaped { + kill_tree(&mut self.child); + let _ = self.child.wait(); + } + } +} + +#[cfg(test)] +thread_local! { + /// Test-only: the reader start (0 for stdout, 1 for stderr) that fails. + static FAIL_READER: std::cell::Cell> = const { std::cell::Cell::new(None) }; + static READERS_STARTED: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + +fn spawn_reader(source: Pipe, limit: usize) -> std::io::Result> +where + Pipe: Read, +{ + #[cfg(test)] + { + let index = READERS_STARTED.with(|started| started.replace(started.get() + 1)); + if FAIL_READER.with(std::cell::Cell::get) == Some(index) { + return Err(std::io::Error::new(std::io::ErrorKind::WouldBlock, "injected")); + } + } + thread::Builder::new().name("jacquard-core-output".into()).spawn(move || bounded_read(source, limit)) +} + +/// The two output readers. Dropping them stops and joins them on Unix, +/// where their reads are interruptible; elsewhere a reader blocked on a +/// pipe a descendant holds is left to finish on its own. +struct Readers { + stop: Arc, + stdout: Option>, + stderr: Option>, +} + +impl Readers { + /// Both outputs, if both close before `deadline`. + fn finish(mut self, deadline: Instant) -> Option<(Output, Output)> { + let done = |reader: &Option>| reader.as_ref().is_none_or(thread::JoinHandle::is_finished); + while !(done(&self.stdout) && done(&self.stderr)) { + if Instant::now() >= deadline { + return None; + } + thread::sleep(Duration::from_millis(10)); + } + let join = |reader: Option>| reader.and_then(|reader| reader.join().ok()); + Some((join(self.stdout.take())?, join(self.stderr.take())?)) + } +} + +impl Drop for Readers { + fn drop(&mut self) { + self.stop.store(true, Ordering::SeqCst); + for reader in [self.stdout.take(), self.stderr.take()].into_iter().flatten() { + if cfg!(unix) || reader.is_finished() { + let _ = reader.join(); + } + } + } +} + +/// The SHA-256 digest of a file's bytes. +pub(crate) fn file_digest(path: &Path) -> std::io::Result { + Ok(hex_digest(&std::fs::read(path)?)) +} + +/// Digests every regular file under `root`, keyed by `/`-separated relative +/// path, refusing other entry kinds. +pub(crate) fn tree(root: &Path) -> Result, Error> { + fn walk(root: &Path, directory: &Path, files: &mut BTreeMap) -> Result<(), Error> { + let io = |error: std::io::Error| Error::preparation(format!("cannot read {}: {error}", directory.display())); + let mut entries = std::fs::read_dir(directory).map_err(io)?.collect::, _>>().map_err(io)?; + entries.sort_by_key(|entry| entry.file_name()); + for entry in entries { + let path = entry.path(); + let kind = entry.file_type().map_err(io)?; + let relative = path.strip_prefix(root).unwrap_or(&path); + let relative = relative.iter().map(|part| part.to_string_lossy()).collect::>().join("/"); + if kind.is_dir() { + walk(root, &path, files)?; + } else if kind.is_file() { + files.insert(relative, file_digest(&path).map_err(io)?); + } else { + return Err(Error::preparation(format!("unsupported filesystem entry in prepared artifacts: {relative}"))); + } + } + Ok(()) + } + let mut files = BTreeMap::new(); + walk(root, root, &mut files)?; + Ok(files) +} + +/// Canonical JSON for a string map: keys in code-point order. +pub(crate) fn canonical(map: &BTreeMap) -> String { + let object = map.iter().map(|(key, value)| (key.clone(), Json::Str(value.clone()))).collect(); + stringify(&Json::Object(object), 4).expect("a flat object") +} + +/// The digest of a string map's canonical JSON. +pub(crate) fn map_digest(map: &BTreeMap) -> String { + hex_digest(canonical(map).as_bytes()) +} + +/// The digest of bytes. +pub(crate) fn digest(data: &[u8]) -> String { + hex_digest(data) +} + +/// Finds an executable the way a shell would, then resolves symbolic links. +pub(crate) fn executable(binary: &str) -> Result { + let missing = || Error::preparation(format!("cannot find Core executable {binary}; configure JACQUARD_BIN using the setup guide")); + let candidate = if binary.contains(std::path::MAIN_SEPARATOR) || binary.contains('/') { + Some(PathBuf::from(binary)) + } else { + // Like Go's exec.LookPath, relative entries (including empty ones, + // which mean the current directory) are not searched. + std::env::var_os("PATH").and_then(|paths| { + std::env::split_paths(&paths) + .filter(|directory| directory.is_absolute()) + .map(|directory| directory.join(binary)) + .find(|path| is_executable(path)) + }) + }; + let candidate = candidate.filter(|path| is_executable(path)).ok_or_else(missing)?; + candidate.canonicalize().map_err(|error| Error::preparation(format!("cannot resolve Core executable {binary}: {error}"))) +} + +#[cfg(unix)] +fn is_executable(path: &Path) -> bool { + use std::os::unix::fs::PermissionsExt; + path.metadata().is_ok_and(|metadata| metadata.is_file() && metadata.permissions().mode() & 0o111 != 0) +} + +#[cfg(not(unix))] +fn is_executable(path: &Path) -> bool { + path.is_file() +} + +/// Standard base64 with padding. +pub(crate) fn base64(data: &[u8]) -> String { + const ALPHABET: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + let mut out = String::with_capacity(data.len().div_ceil(3) * 4); + for chunk in data.chunks(3) { + let bytes = [chunk[0], chunk.get(1).copied().unwrap_or(0), chunk.get(2).copied().unwrap_or(0)]; + let n = u32::from(bytes[0]) << 16 | u32::from(bytes[1]) << 8 | u32::from(bytes[2]); + for (index, shift) in [18, 12, 6, 0].into_iter().enumerate() { + out.push(if index <= chunk.len() { char::from(ALPHABET[(n >> shift) as usize & 63]) } else { '=' }); + } + } + out +} + +#[cfg(all(test, unix))] +mod unix_tests { + use super::{FAIL_READER, READERS_STARTED, run}; + use std::ffi::OsStr; + use std::path::Path; + use std::time::{Duration, Instant}; + + #[test] + fn a_reader_that_cannot_start_stops_the_other_and_the_command() { + for position in [0, 1] { + let marker = std::env::temp_dir().join(format!("jacquard-host-rust-reader-{}-{position}", std::process::id())); + let _ = std::fs::remove_file(&marker); + READERS_STARTED.with(|started| started.set(0)); + FAIL_READER.with(|fail| fail.set(Some(position))); + let script = format!("sleep 0.3; touch '{}'", marker.display()); + let started = Instant::now(); + let error = run(Path::new("/bin/sh"), Path::new("/"), &[OsStr::new("-c"), OsStr::new(&script)]).unwrap_err(); + FAIL_READER.with(|fail| fail.set(None)); + assert!(error.to_string().contains("cannot start a reader: injected"), "{error}"); + assert!(started.elapsed() < Duration::from_millis(250), "cleanup waited for the command"); + std::thread::sleep(Duration::from_millis(600)); + assert!(!marker.exists(), "the command kept running after reader {position} failed to start"); + } + } +} + +#[cfg(test)] +mod tests { + use super::base64; + + #[test] + fn base64_matches_rfc_4648_vectors() { + for (input, expected) in + [("", ""), ("f", "Zg=="), ("fo", "Zm8="), ("foo", "Zm9v"), ("foob", "Zm9vYg=="), ("fooba", "Zm9vYmE="), ("foobar", "Zm9vYmFy")] + { + assert_eq!(base64(input.as_bytes()), expected); + } + } +} diff --git a/rust/src/sdk/data.rs b/rust/src/sdk/data.rs new file mode 100644 index 0000000..b856a4e --- /dev/null +++ b/rust/src/sdk/data.rs @@ -0,0 +1,763 @@ +//! Native Rust values and the prepared program's Jacquard data. +//! +//! [`Encode`] and [`Decode`] are implemented for `bool`, the integer types, +//! `f64`, text, [`Hash`], [`Value`], [`Dynamic`], `()`, tuples, `Vec` (a +//! `List`), and `Option`. Application types implement them with the +//! [`record!`](crate::record) and [`variants!`](crate::variants) macros, or +//! by hand with an [`Encoder`] and a [`Decoder`]. + +use std::collections::BTreeMap; +use std::fmt; + +use crate::json::{INT_MAX, INT_MIN}; +use crate::protocol::{BoundaryType, Hash, Limits, Value}; + +use super::identities::{Group, display_to_store}; +use super::{Error, InvalidArgument, MappingError}; + +const BUILTINS: [&str; 6] = ["true", "false", "some", "none", "cons", "nil"]; + +/// A native value with no application type: Jacquard data by public name. +#[derive(Clone, Debug, PartialEq)] +pub enum Dynamic { + Int(i64), + Real(f64), + Text(String), + Bool(bool), + Hash(Hash), + /// A positional product; the empty tuple is unit. + Tuple(Vec), + List(Vec), + Option(Option>), + /// A constructor by public name (store spelling such as `stock`, or + /// display spelling such as `Stock`) and its positional fields. + Variant(String, Vec), +} + +impl Dynamic { + /// A constructor value by public name. + pub fn variant(name: impl Into, fields: impl IntoIterator) -> Dynamic { + Dynamic::Variant(name.into(), fields.into_iter().collect()) + } + + /// Unit, the empty tuple. + pub fn unit() -> Dynamic { + Dynamic::Tuple(Vec::new()) + } +} + +impl From for Dynamic { + fn from(value: i64) -> Self { + Dynamic::Int(value) + } +} + +impl From for Dynamic { + fn from(value: f64) -> Self { + Dynamic::Real(value) + } +} + +impl From for Dynamic { + fn from(value: bool) -> Self { + Dynamic::Bool(value) + } +} + +impl From<&str> for Dynamic { + fn from(value: &str) -> Self { + Dynamic::Text(value.to_owned()) + } +} + +impl From for Dynamic { + fn from(value: String) -> Self { + Dynamic::Text(value) + } +} + +/// Converts a native value to a boundary value. +pub trait Encode { + /// Encodes `self`; `cx` names the value's location for error messages. + fn encode(&self, cx: &Encoder<'_>) -> Result; +} + +/// Converts a boundary value to a native value. +pub trait Decode: Sized { + /// Decodes `value`. + fn decode(cx: &Decoder<'_>, value: &Value) -> Result; +} + +/// The prepared program's public names and identities, used to map native +/// values. It is immutable once the program is prepared. +#[derive(Debug)] +pub struct Data { + names: BTreeMap, + constructors: BTreeMap, + labels: BTreeMap, + primitives: BTreeMap, + /// Each declaration group's head identity and its member identities. + members: BTreeMap>, + limits: Limits, +} + +impl Data { + pub(crate) fn new(names: BTreeMap, constructors: BTreeMap, groups: &[Group], limits: Limits) -> Data { + let labels = constructors.iter().map(|(name, id)| (id.clone(), name.clone())).collect(); + let primitives = ["int", "real", "text", "bool", "hash", "list", "option"] + .into_iter() + .filter_map(|name| names.get(name).map(|id| (id.clone(), name))) + .collect(); + let members = groups + .iter() + .filter_map(|group| group.split_first()) + .map(|(head, rest)| (head.hash.clone(), rest.iter().map(|entry| entry.hash.clone()).collect())) + .collect(); + Data { names, constructors, labels, primitives, members, limits } + } + + /// The session limits that bound mapped values. + pub fn limits(&self) -> &Limits { + &self.limits + } + + /// The exact identity of a public type, constructor, effect, or term name + /// in the prepared program and its discovered prelude. + pub fn identity(&self, name: &str) -> Result { + self.names.get(&display_to_store(name)).cloned().ok_or_else(|| Error::binding(format!("unknown name {name}"))) + } + + /// The public name of a constructor or operation identity. + pub fn label(&self, identity: &Hash) -> Option<&str> { + self.labels.get(identity).map(String::as_str) + } + + pub(crate) fn name(&self, name: &str) -> Option<&Hash> { + self.names.get(name) + } + + fn constructor_id(&self, name: &str) -> Option<&Hash> { + self.constructors.get(&display_to_store(name)) + } + + /// A [`Dynamic::Variant`] after checking that the constructor exists. + pub fn make(&self, name: &str, fields: impl IntoIterator) -> Result { + if self.constructor_id(name).is_none() { + return Err(Error::binding(format!("unknown constructor {name}"))); + } + Ok(Dynamic::Variant(display_to_store(name), fields.into_iter().collect())) + } + + /// Encodes a native value. + pub fn encode(&self, value: &T) -> Result { + Ok(value.encode(&Encoder::root(self, "value".into(), 3)?)?) + } + + /// Decodes a boundary value into a native value. + pub fn decode(&self, value: &Value) -> Result { + T::decode(&Decoder { data: self }, value).map_err(|error| Error::binding(error.message)) + } + + pub(crate) fn primitive(&self, expected: &BoundaryType) -> Option<&'static str> { + match expected { + BoundaryType::Nominal(identity, _) => self.primitives.get(identity).copied(), + BoundaryType::Tuple(_) => None, + } + } + + fn is_nullary(&self, value: &Value, name: &str) -> bool { + matches!(value, Value::Constructor(id, fields) if fields.is_empty() && self.label(id) == Some(name)) + } + + /// Walks a proper List, bounded by the collection limit. + fn list_items<'v>(&self, value: &'v Value) -> Result, MappingError> { + let mut items = Vec::new(); + let mut cursor = value; + loop { + match cursor { + Value::Constructor(id, fields) if fields.len() == 2 && self.label(id) == Some("cons") => { + if items.len() >= self.limits.max_collection_items { + return Err(MappingError::new("list exceeds the selected limit")); + } + items.push(&fields[0]); + cursor = &fields[1]; + } + Value::Constructor(id, fields) if fields.is_empty() && self.label(id) == Some("nil") => return Ok(items), + _ => return Err(MappingError::new("expected a proper List")), + } + } + } + + /// Checks that an encoded value has the checked boundary type, as far as + /// the prepared metadata describes it: primitives, Lists, Options, and + /// tuples exactly, and other data by its constructor's declaration group. + pub(crate) fn conform(&self, value: &Value, expected: &BoundaryType, path: &str) -> Result<(), InvalidArgument> { + let refuse = |what: &str| Err(InvalidArgument::at(path, format!("expected {what}"))); + let child = |expected: &BoundaryType| match expected { + BoundaryType::Nominal(_, arguments) => arguments.first().cloned(), + BoundaryType::Tuple(_) => None, + }; + match (self.primitive(expected), expected) { + (_, BoundaryType::Tuple(items)) => match value { + Value::Tuple(values) if values.len() == items.len() => { + for (index, (value, item)) in values.iter().zip(items).enumerate() { + self.conform(value, item, &format!("{path}[{index}]"))?; + } + Ok(()) + } + _ => refuse(&format!("a {}-item tuple", items.len())), + }, + (Some("int"), _) => { + if matches!(value, Value::Int(_)) { + Ok(()) + } else { + refuse("an integer") + } + } + (Some("real"), _) => { + if matches!(value, Value::Real(_)) { + Ok(()) + } else { + refuse("a Real (f64)") + } + } + (Some("text"), _) => { + if matches!(value, Value::Text(_)) { + Ok(()) + } else { + refuse("text") + } + } + (Some("hash"), _) => { + if matches!(value, Value::Hash(_)) { + Ok(()) + } else { + refuse("a Hash") + } + } + (Some("bool"), _) => { + if self.is_nullary(value, "true") || self.is_nullary(value, "false") { + Ok(()) + } else { + refuse("a bool") + } + } + (Some("list"), _) => { + let items = self.list_items(value).map_err(|_| InvalidArgument::at(path, "expected a List (Vec)"))?; + match child(expected) { + Some(item) => { + for (index, value) in items.into_iter().enumerate() { + self.conform(value, &item, &format!("{path}[{index}]"))?; + } + Ok(()) + } + None => Ok(()), + } + } + (Some("option"), _) => match value { + _ if self.is_nullary(value, "none") => Ok(()), + Value::Constructor(id, fields) if fields.len() == 1 && self.label(id) == Some("some") => match child(expected) { + Some(inner) => self.conform(&fields[0], &inner, path), + None => Ok(()), + }, + _ => refuse("an Option"), + }, + (_, BoundaryType::Nominal(identity, _)) => match (value, self.members.get(identity)) { + (Value::Constructor(id, _), Some(members)) if members.contains(id) => Ok(()), + (Value::Constructor(_, _), None) => Ok(()), + _ => refuse(&format!("a {} value", self.label_of_type(identity))), + }, + } + } + + fn label_of_type(&self, identity: &Hash) -> String { + self.names.iter().find(|(_, id)| *id == identity).map_or_else(|| identity.to_string(), |(name, _)| name.clone()) + } +} + +/// Encodes one value at a location, bounded by the selected JSON depth. +pub struct Encoder<'a> { + data: &'a Data, + path: String, + depth: usize, +} + +impl<'a> Encoder<'a> { + pub(crate) fn root(data: &'a Data, path: String, depth: usize) -> Result, InvalidArgument> { + let encoder = Encoder { data, path, depth }; + encoder.check_depth(depth)?; + Ok(encoder) + } + + fn check_depth(&self, depth: usize) -> Result<(), InvalidArgument> { + if depth > self.data.limits.max_json_depth { + return Err(self.invalid("value exceeds the v0 JSON depth; use smaller independent batches")); + } + Ok(()) + } + + /// The program's data. + pub fn data(&self) -> &'a Data { + self.data + } + + /// The location being encoded, such as `triage.batch.argument[1][0].service`. + pub fn path(&self) -> &str { + &self.path + } + + /// An error at this location. + pub fn invalid(&self, message: impl fmt::Display) -> InvalidArgument { + InvalidArgument::at(&self.path, message) + } + + fn child(&self, segment: fmt::Arguments<'_>, extra: usize) -> Result, InvalidArgument> { + let child = Encoder { data: self.data, path: format!("{}{segment}", self.path), depth: self.depth + extra }; + child.check_depth(child.depth)?; + Ok(child) + } + + /// Encodes a named field of a record or variant. + pub fn field(&self, name: &str, value: &T) -> Result { + value.encode(&self.child(format_args!(".{name}"), 2)?) + } + + /// Encodes a positional item of a tuple. + pub fn item(&self, index: usize, value: &T) -> Result { + value.encode(&self.child(format_args!("[{index}]"), 2)?) + } + + /// A constructor value by public name, such as `"Job"` or `"job"`. + /// Bool, List, and Option constructors are refused: use the native + /// `bool`, `Vec`, and `Option` forms. + pub fn constructor(&self, name: &str, fields: Vec) -> Result { + if BUILTINS.contains(&display_to_store(name).as_str()) { + return Err(self.invalid(format!("use the native form of builtin constructor {name}"))); + } + self.builtin(name, fields) + } + + fn builtin(&self, name: &str, fields: Vec) -> Result { + match self.data.constructor_id(name) { + Some(id) => Ok(Value::Constructor(id.clone(), fields)), + None => Err(self.invalid(format!("unknown constructor {name}"))), + } + } + + fn list(&self, items: &[T]) -> Result { + if items.len() > self.data.limits.max_collection_items { + return Err(self.invalid("list exceeds max_collection_items")); + } + // A linked list nests one constructor per element on the wire. + self.check_depth(self.depth.saturating_add(2 * items.len()))?; + let mut result = self.builtin("nil", Vec::new())?; + for (index, item) in items.iter().enumerate().rev() { + let value = item.encode(&self.child(format_args!("[{index}]"), 2 * (index + 1))?)?; + result = self.builtin("cons", vec![value, result])?; + } + Ok(result) + } + + fn option(&self, value: Option<&T>) -> Result { + match value { + None => self.builtin("none", Vec::new()), + Some(inner) => { + let inner = inner.encode(&self.child(format_args!(""), 2)?)?; + self.builtin("some", vec![inner]) + } + } + } + + fn text(&self, text: &str) -> Result { + if text.len() > self.data.limits.max_text_bytes { + return Err(self.invalid("text exceeds max_text_bytes")); + } + Ok(Value::Text(text.to_owned())) + } + + fn int(&self, value: i128) -> Result { + if value < i128::from(INT_MIN) || value > i128::from(INT_MAX) { + return Err(self.invalid("integer is outside Core's 63-bit Int range")); + } + Ok(Value::Int(value as i64)) + } +} + +/// Decodes boundary values with the program's data. +pub struct Decoder<'a> { + data: &'a Data, +} + +/// The positional fields of one constructor value, taken in order. +pub struct Fields<'a, 'v> { + decoder: &'a Decoder<'a>, + fields: std::slice::Iter<'v, Value>, +} + +impl<'a> Decoder<'a> { + /// The program's data. + pub fn data(&self) -> &'a Data { + self.data + } + + /// Decodes a nested value. + pub fn decode(&self, value: &Value) -> Result { + T::decode(self, value) + } + + /// Whether `value` is built by the named constructor. + pub fn is(&self, value: &Value, constructor: &str) -> bool { + matches!((value, self.data.constructor_id(constructor)), (Value::Constructor(id, _), Some(expected)) if id == expected) + } + + /// The fields of a value built by the named constructor with `count` + /// fields. + pub fn fields<'v>(&'a self, value: &'v Value, constructor: &str, count: usize) -> Result, MappingError> { + match value { + Value::Constructor(_, fields) if self.is(value, constructor) && fields.len() == count => { + Ok(Fields { decoder: self, fields: fields.iter() }) + } + _ => Err(MappingError::new(format!("{constructor} result has the wrong constructor or field count"))), + } + } + + /// An error for a value that is none of a type's constructors. + pub fn mismatch(&self, type_name: &str) -> MappingError { + MappingError::new(format!("unknown {type_name} case")) + } +} + +impl Fields<'_, '_> { + /// Decodes the next field. + pub fn take(&mut self) -> Result { + match self.fields.next() { + Some(value) => T::decode(self.decoder, value), + None => Err(MappingError::new("constructor has too few fields")), + } + } +} + +impl Encode for &T { + fn encode(&self, cx: &Encoder<'_>) -> Result { + (**self).encode(cx) + } +} + +impl Encode for Box { + fn encode(&self, cx: &Encoder<'_>) -> Result { + (**self).encode(cx) + } +} + +impl Decode for Box { + fn decode(cx: &Decoder<'_>, value: &Value) -> Result { + T::decode(cx, value).map(Box::new) + } +} + +impl Encode for bool { + fn encode(&self, cx: &Encoder<'_>) -> Result { + cx.builtin(if *self { "true" } else { "false" }, Vec::new()) + } +} + +impl Decode for bool { + fn decode(cx: &Decoder<'_>, value: &Value) -> Result { + match value { + _ if cx.data.is_nullary(value, "true") => Ok(true), + _ if cx.data.is_nullary(value, "false") => Ok(false), + _ => Err(MappingError::new("expected Bool")), + } + } +} + +macro_rules! integers { + ($($t:ty),*) => {$( + impl Encode for $t { + fn encode(&self, cx: &Encoder<'_>) -> Result { + cx.int(*self as i128) + } + } + + impl Decode for $t { + fn decode(_: &Decoder<'_>, value: &Value) -> Result { + match value { + Value::Int(n) => <$t>::try_from(*n).map_err(|_| MappingError::new(concat!("expected Int that fits ", stringify!($t)))), + _ => Err(MappingError::new(concat!("expected Int that fits ", stringify!($t)))), + } + } + } + )*}; +} + +integers!(i8, i16, i32, i64, isize, u8, u16, u32, u64, usize); + +impl Encode for f64 { + fn encode(&self, _: &Encoder<'_>) -> Result { + Ok(Value::real(*self)) + } +} + +impl Encode for f32 { + fn encode(&self, _: &Encoder<'_>) -> Result { + Ok(Value::real(f64::from(*self))) + } +} + +impl Decode for f64 { + fn decode(_: &Decoder<'_>, value: &Value) -> Result { + match value { + Value::Real(bits) => Ok(f64::from_bits(*bits)), + _ => Err(MappingError::new("expected Real")), + } + } +} + +impl Encode for str { + fn encode(&self, cx: &Encoder<'_>) -> Result { + cx.text(self) + } +} + +impl Encode for String { + fn encode(&self, cx: &Encoder<'_>) -> Result { + cx.text(self) + } +} + +impl Decode for String { + fn decode(_: &Decoder<'_>, value: &Value) -> Result { + match value { + Value::Text(text) => Ok(text.clone()), + _ => Err(MappingError::new("expected Text")), + } + } +} + +impl Encode for Hash { + fn encode(&self, _: &Encoder<'_>) -> Result { + Ok(Value::Hash(self.clone())) + } +} + +impl Decode for Hash { + fn decode(_: &Decoder<'_>, value: &Value) -> Result { + match value { + Value::Hash(hash) => Ok(hash.clone()), + _ => Err(MappingError::new("expected Hash")), + } + } +} + +/// A raw boundary value passes through unchanged. +impl Encode for Value { + fn encode(&self, _: &Encoder<'_>) -> Result { + Ok(self.clone()) + } +} + +impl Decode for Value { + fn decode(_: &Decoder<'_>, value: &Value) -> Result { + Ok(value.clone()) + } +} + +impl Encode for [T] { + fn encode(&self, cx: &Encoder<'_>) -> Result { + cx.list(self) + } +} + +impl Encode for Vec { + fn encode(&self, cx: &Encoder<'_>) -> Result { + cx.list(self) + } +} + +impl Decode for Vec { + fn decode(cx: &Decoder<'_>, value: &Value) -> Result { + cx.data.list_items(value)?.into_iter().map(|item| T::decode(cx, item)).collect() + } +} + +impl Encode for Option { + fn encode(&self, cx: &Encoder<'_>) -> Result { + cx.option(self.as_ref()) + } +} + +impl Decode for Option { + fn decode(cx: &Decoder<'_>, value: &Value) -> Result { + match value { + _ if cx.data.is_nullary(value, "none") => Ok(None), + Value::Constructor(id, fields) if fields.len() == 1 && cx.data.label(id) == Some("some") => T::decode(cx, &fields[0]).map(Some), + _ => Err(MappingError::new("expected Option")), + } + } +} + +macro_rules! tuples { + ($(($($name:ident $index:tt),*)),*) => {$( + impl<$($name: Encode),*> Encode for ($($name,)*) { + #[allow(unused_variables)] + fn encode(&self, cx: &Encoder<'_>) -> Result { + Ok(Value::Tuple(vec![$(cx.item($index, &self.$index)?),*])) + } + } + + impl<$($name: Decode),*> Decode for ($($name,)*) { + #[allow(unused_variables, unused_mut)] + fn decode(cx: &Decoder<'_>, value: &Value) -> Result { + const COUNT: usize = <[&str]>::len(&[$(stringify!($name)),*]); + match value { + Value::Tuple(items) if items.len() == COUNT => { + let mut items = items.iter(); + Ok(($($name::decode(cx, items.next().expect("counted"))?,)*)) + } + _ => Err(MappingError::new("tuple shape mismatch")), + } + } + } + )*}; +} + +tuples!((), (A 0), (A 0, B 1), (A 0, B 1, C 2), (A 0, B 1, C 2, D 3), (A 0, B 1, C 2, D 3, E 4), (A 0, B 1, C 2, D 3, E 4, F 5)); + +impl Encode for Dynamic { + fn encode(&self, cx: &Encoder<'_>) -> Result { + match self { + Dynamic::Int(n) => n.encode(cx), + Dynamic::Real(r) => r.encode(cx), + Dynamic::Text(text) => text.encode(cx), + Dynamic::Bool(b) => b.encode(cx), + Dynamic::Hash(hash) => hash.encode(cx), + Dynamic::Tuple(items) => { + Ok(Value::Tuple(items.iter().enumerate().map(|(index, item)| cx.item(index, item)).collect::>()?)) + } + Dynamic::List(items) => cx.list(items), + Dynamic::Option(inner) => cx.option(inner.as_deref()), + Dynamic::Variant(name, fields) => { + let fields = fields.iter().enumerate().map(|(index, field)| field.encode(&cx.child(format_args!(".{index}"), 2)?)); + let fields = fields.collect::>()?; + cx.constructor(name, fields) + } + } + } +} + +impl Decode for Dynamic { + fn decode(cx: &Decoder<'_>, value: &Value) -> Result { + Ok(match value { + Value::Int(n) => Dynamic::Int(*n), + Value::Real(bits) => Dynamic::Real(f64::from_bits(*bits)), + Value::Text(text) => Dynamic::Text(text.clone()), + Value::Hash(hash) => Dynamic::Hash(hash.clone()), + Value::Tuple(items) => Dynamic::Tuple(items.iter().map(|item| Dynamic::decode(cx, item)).collect::>()?), + Value::Constructor(id, fields) => { + let Some(label) = cx.data.label(id) else { + return Err(MappingError::new(format!("unmapped constructor {id}"))); + }; + match label { + "true" | "false" => Dynamic::Bool(bool::decode(cx, value)?), + "cons" | "nil" => Dynamic::List(Vec::decode(cx, value)?), + "some" | "none" => Dynamic::Option(Option::::decode(cx, value)?.map(Box::new)), + _ => { + Dynamic::Variant(label.to_owned(), fields.iter().map(|field| Dynamic::decode(cx, field)).collect::>()?) + } + } + } + }) + } +} + +/// Implements [`Encode`] and [`Decode`] for a struct as one Jacquard +/// constructor. The listed fields, in order, are the constructor's +/// positional fields. +/// +/// ``` +/// struct Job { +/// id: String, +/// effort: i64, +/// } +/// +/// jacquard_host::record!(Job = "Job" { id, effort }); +/// ``` +#[macro_export] +macro_rules! record { + ($ty:ty = $constructor:literal { $($field:ident),* $(,)? }) => { + impl $crate::sdk::Encode for $ty { + fn encode(&self, cx: &$crate::sdk::Encoder<'_>) -> ::std::result::Result<$crate::protocol::Value, $crate::sdk::InvalidArgument> { + cx.constructor($constructor, ::std::vec![$(cx.field(::std::stringify!($field), &self.$field)?),*]) + } + } + + impl $crate::sdk::Decode for $ty { + #[allow(unused_mut)] + fn decode(cx: &$crate::sdk::Decoder<'_>, value: &$crate::protocol::Value) -> ::std::result::Result { + let count = <[&str]>::len(&[$(::std::stringify!($field)),*]); + let mut fields = cx.fields(value, $constructor, count)?; + ::std::result::Result::Ok(Self { $($field: fields.take()?),* }) + } + } + }; +} + +/// Implements [`Encode`] and [`Decode`] for an enum whose variants are the +/// constructors of one Jacquard data type. Unit variants are nullary +/// constructors; tuple and struct variants list their fields in order. +/// +/// ``` +/// struct Plan { +/// score: i64, +/// } +/// jacquard_host::record!(Plan = "Plan" { score }); +/// +/// enum Planning { +/// Planned { plan: Plan }, +/// InvalidPlan(String), +/// } +/// +/// jacquard_host::variants!(Planning { +/// Planned { plan } = "Planned", +/// InvalidPlan(reason) = "InvalidPlan", +/// }); +/// +/// enum Urgency { +/// Page, +/// Today, +/// } +/// +/// jacquard_host::variants!(Urgency { Page = "PageNow", Today = "WorkToday" }); +/// ``` +#[macro_export] +macro_rules! variants { + ($ty:ty { $($variant:ident $(( $($positional:ident),* $(,)? ))? $({ $($named:ident),* $(,)? })? = $constructor:literal),* $(,)? }) => { + impl $crate::sdk::Encode for $ty { + fn encode(&self, cx: &$crate::sdk::Encoder<'_>) -> ::std::result::Result<$crate::protocol::Value, $crate::sdk::InvalidArgument> { + match self { + $(Self::$variant $(( $($positional),* ))? $({ $($named),* })? => cx.constructor( + $constructor, + ::std::vec![ + $($(cx.field(::std::stringify!($positional), $positional)?),*)? + $($(cx.field(::std::stringify!($named), $named)?),*)? + ], + ),)* + } + } + } + + impl $crate::sdk::Decode for $ty { + #[allow(unused_mut, unused_variables)] + fn decode(cx: &$crate::sdk::Decoder<'_>, value: &$crate::protocol::Value) -> ::std::result::Result { + $(if cx.is(value, $constructor) { + let count = <[&str]>::len(&[$($(::std::stringify!($positional)),*)? $($(::std::stringify!($named)),*)?]); + let mut fields = cx.fields(value, $constructor, count)?; + return ::std::result::Result::Ok( + Self::$variant $(( $({ let _ = ::std::stringify!($positional); fields.take()? }),* ))? $({ $($named: fields.take()?),* })? + ); + })* + ::std::result::Result::Err(cx.mismatch(::std::stringify!($ty))) + } + } + }; +} diff --git a/rust/src/sdk/identities.rs b/rust/src/sdk/identities.rs new file mode 100644 index 0000000..dc716d5 --- /dev/null +++ b/rust/src/sdk/identities.rs @@ -0,0 +1,220 @@ +//! Public names and identities printed by Core's `hash` and `check +//! --print-sigs` commands. + +use std::collections::{BTreeMap, BTreeSet}; + +use crate::protocol::{BoundaryType, Hash, Interface}; + +use super::Error; + +/// One public name and identity printed by `jacquard hash`. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct Entry { + pub name: String, + pub hash: Hash, +} + +/// One declaration group: a type or effect identity followed by its +/// constructors or operations, in Core's printed order. +pub(crate) type Group = Vec; + +/// Parses one `N:name hash` line. +fn hash_line(line: &str) -> Option<(&str, Entry)> { + let (index, rest) = line.split_once(':')?; + let (name, hash) = rest.split_once(' ')?; + if index.is_empty() || !index.bytes().all(|byte| byte.is_ascii_digit()) || name.is_empty() || name.contains(char::is_whitespace) { + return None; + } + let hash = Hash::parse(hash).ok()?; + Some((index, Entry { name: name.to_owned(), hash })) +} + +/// Parses `jacquard hash` output into declaration groups. +pub(crate) fn hash_groups(output: &str) -> Result, Error> { + let mut order: Vec<&str> = Vec::new(); + let mut groups: BTreeMap<&str, Group> = BTreeMap::new(); + for line in output.split('\n') { + let Some((index, entry)) = hash_line(line.trim()) else { continue }; + if !groups.contains_key(index) { + order.push(index); + } + groups.entry(index).or_default().push(entry); + } + if order.is_empty() { + return Err(Error::binding("Core hash command returned no public identities")); + } + Ok(order.into_iter().map(|index| groups.remove(index).unwrap_or_default()).collect()) +} + +/// Refuses a public name that denotes different identities in different +/// declaration groups. +pub(crate) fn check_groups(groups: &[Group]) -> Result<(), Error> { + let mut seen: BTreeMap<&str, BTreeSet<&Hash>> = BTreeMap::new(); + for group in groups { + let mut current: BTreeMap<&str, BTreeSet<&Hash>> = BTreeMap::new(); + for entry in group { + current.entry(&entry.name).or_default().insert(&entry.hash); + } + for (name, hashes) in current { + let previous = seen.entry(name).or_default(); + if !previous.is_empty() && !hashes.is_subset(previous) { + return Err(Error::binding(format!( + "ambiguous public name {name} occurs in different declaration groups; use distinct names or explicit low-level identities" + ))); + } + previous.extend(hashes); + } + } + Ok(()) +} + +/// Converts a display name such as `WorkToday` to its store spelling, +/// `work-today`; lowercase names are unchanged. +pub(crate) fn display_to_store(name: &str) -> String { + if !name.starts_with(|c: char| c.is_uppercase()) { + return name.to_owned(); + } + let mut out = String::new(); + for (index, c) in name.char_indices() { + if index > 0 && c.is_uppercase() { + out.push('-'); + } + out.extend(c.to_lowercase()); + } + out +} + +/// The form used to look up effects and operations: store spelling, with +/// Rust's `_` accepted for `-`. +pub(crate) fn normal(name: &str) -> String { + display_to_store(name).replace('_', "-") +} + +/// Tokens of a printed signature. +fn tokens(text: &str) -> Result, String> { + let mut tokens = Vec::new(); + let mut rest = text.trim_start(); + while !rest.is_empty() { + let length = if rest.starts_with("->{") { + 3 + } else if rest.starts_with(['(', ')', ',', '{', '}']) { + 1 + } else if rest.starts_with(|c: char| c.is_ascii_alphabetic()) { + rest.find(|c: char| !(c.is_ascii_alphanumeric() || matches!(c, '.' | '?' | '!' | '-'))).unwrap_or(rest.len()) + } else { + return Err(format!("unsupported signature near {:?}", rest.chars().take(12).collect::())); + }; + tokens.push(&rest[..length]); + rest = rest[length..].trim_start(); + } + Ok(tokens) +} + +/// Reads a closed v0 signature printed by `check --print-sigs`, such as +/// `(List Line, Int) ->{Inventory} Option Text`. Polymorphic and +/// effect-polymorphic signatures are refused. +pub(crate) fn parse_signature(text: &str, names: &BTreeMap) -> Result { + let mut parser = Parser { tokens: tokens(text)?, index: 0, names }; + parser.parse() +} + +struct Parser<'a> { + tokens: Vec<&'a str>, + index: usize, + names: &'a BTreeMap, +} + +impl Parser<'_> { + fn peek(&self) -> &str { + self.tokens.get(self.index).copied().unwrap_or("") + } + + fn take(&mut self, expected: &str) -> Result { + let token = self.peek().to_owned(); + if token.is_empty() || (!expected.is_empty() && token != expected) { + return Err(format!("expected {expected:?}, found {token:?}")); + } + self.index += 1; + Ok(token) + } + + fn identity(&self, display: &str) -> Result { + self.names + .get(&display_to_store(display)) + .or_else(|| self.names.get(display)) + .cloned() + .ok_or_else(|| format!("no identity for {display}")) + } + + fn list(&mut self, close: &str) -> Result, String> { + let mut out = Vec::new(); + if self.peek() == close { + self.index += 1; + return Ok(out); + } + loop { + out.push(self.typ()?); + if self.peek() == "," { + self.index += 1; + if self.peek() == close { + self.index += 1; + return Ok(out); + } + continue; + } + self.take(close)?; + return Ok(out); + } + } + + fn atom(&mut self) -> Result { + let token = self.take("")?; + if token == "(" { + let items = self.list(")")?; + if items.len() == 1 && self.tokens[self.index - 2] != "," { + return Ok(items.into_iter().next().expect("one item")); + } + return Ok(BoundaryType::Tuple(items)); + } + if !token.starts_with(|c: char| c.is_alphabetic()) { + return Err(format!("unsupported type {token:?}")); + } + Ok(BoundaryType::Nominal(self.identity(&token)?, Vec::new())) + } + + fn typ(&mut self) -> Result { + let head = self.atom()?; + let BoundaryType::Nominal(identity, mut arguments) = head else { return Ok(head) }; + while !matches!(self.peek(), "" | "," | ")" | "}" | "->{") { + arguments.push(self.atom()?); + } + Ok(BoundaryType::Nominal(identity, arguments)) + } + + fn parse(&mut self) -> Result { + self.take("(")?; + let parameters = self.list(")")?; + self.take("->{")?; + let mut effects = Vec::new(); + if self.peek() != "}" { + loop { + let name = self.take("")?; + let identity = self.identity(&name)?; + if !effects.contains(&identity) { + effects.push(identity); + } + if self.peek() != "," { + break; + } + self.index += 1; + } + } + self.take("}")?; + let result = self.typ()?; + if !self.peek().is_empty() { + return Err("trailing signature tokens".into()); + } + effects.sort(); + Ok(Interface { parameters, result, effects }) + } +} diff --git a/rust/src/sdk/mod.rs b/rust/src/sdk/mod.rs new file mode 100644 index 0000000..29c20e0 --- /dev/null +++ b/rust/src/sdk/mod.rs @@ -0,0 +1,971 @@ +//! Call checked Jacquard functions from Rust: prepare and cache programs +//! with Core's public commands, map native Rust values, bind Rust closures to +//! exact operation identities, and keep the raw evidence of every +//! invocation. Each invocation runs one fresh, serial worker. +//! +//! ```no_run +//! use jacquard_host::sdk::{PrepareOptions, Runtime, RuntimeOptions}; +//! +//! # fn main() -> Result<(), jacquard_host::sdk::Error> { +//! let runtime = Runtime::new(RuntimeOptions::default())?; +//! let program = runtime.prepare(&["rules.jac"], &PrepareOptions::default())?; +//! let shipping = program.function("shipping-cost")?; +//! let cost: i64 = shipping.call((35,))?; +//! # Ok(()) +//! # } +//! ``` + +mod core; +mod data; +mod identities; +mod operations; +#[cfg(test)] +mod tests; + +use std::collections::{BTreeMap, BTreeSet}; +use std::ffi::OsStr; +use std::fmt; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex, MutexGuard}; +use std::time::Duration; + +use crate::carrier::{Cancel, Options}; +use crate::compat::{CORE_VERSION, PROTOCOL}; +use crate::host::{Failure, Host, HostOptions, InvocationError, InvocationResult, OperationRegistry}; +use crate::json::{Json, Object, decode_object, frame_bytes, stringify}; +use crate::protocol::{BoundaryType, HARD_LIMITS, Hash, Interface, Limits, Value, invoke_frame}; + +pub use data::{Data, Decode, Decoder, Dynamic, Encode, Encoder, Fields}; +pub use operations::{IntoHandler, Operations, WithContext}; + +use identities::{Entry, Group, check_groups, hash_groups, normal, parse_signature}; + +const PROGRAM_SCHEMA: &str = "jacquard-rust-program-v1"; + +/// Prepared programs live here unless [`PrepareOptions::cache_dir`] is set. +pub const DEFAULT_CACHE_DIR: &str = ".scratch/jacquard-host-rust"; + +/// A local input that cannot be sent; no worker started. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct InvalidArgument { + pub message: String, +} + +impl InvalidArgument { + /// An error at a value's location, such as `shipping-cost.argument[0]`. + pub fn at(path: &str, message: impl fmt::Display) -> Self { + InvalidArgument { message: format!("{path}: {message}") } + } +} + +impl fmt::Display for InvalidArgument { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(&self.message) + } +} + +impl std::error::Error for InvalidArgument {} + +/// A boundary value that does not decode into the requested Rust type. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct MappingError { + pub message: String, +} + +impl MappingError { + pub fn new(message: impl Into) -> Self { + MappingError { message: message.into() } + } +} + +impl fmt::Display for MappingError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(&self.message) + } +} + +impl std::error::Error for MappingError {} + +/// An SDK failure. +#[derive(Clone, Debug)] +pub enum Error { + /// The program could not be checked, hashed, installed, or loaded from + /// its cache. + Preparation(String), + /// A mapping or binding problem: an unknown name, a missing handler, or + /// a successful Core result that does not decode into the requested + /// type. `result` is set when Core completed. + Binding { message: String, result: Option> }, + /// A local input that cannot be sent; no worker started. + InvalidArgument(InvalidArgument), + /// The invocation ran and did not succeed; classify it with + /// [`Error::failure`]. + Invocation(InvocationError), +} + +impl Error { + pub(crate) fn preparation(message: impl Into) -> Self { + Error::Preparation(message.into()) + } + + pub(crate) fn binding(message: impl Into) -> Self { + Error::Binding { message: message.into(), result: None } + } + + /// How an invocation failed, for [`Error::Invocation`]. + pub fn failure(&self) -> Option { + match self { + Error::Invocation(error) => Some(error.failure()), + _ => None, + } + } + + /// The raw result, when a worker ran. + pub fn result(&self) -> Option<&InvocationResult> { + match self { + Error::Invocation(error) => Some(&error.0), + Error::Binding { result, .. } => result.as_deref(), + _ => None, + } + } +} + +impl fmt::Display for Error { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Error::Preparation(message) | Error::Binding { message, .. } => f.write_str(message), + Error::InvalidArgument(error) => error.fmt(f), + Error::Invocation(error) => error.fmt(f), + } + } +} + +impl std::error::Error for Error {} + +impl From for Error { + fn from(error: InvalidArgument) -> Self { + Error::InvalidArgument(error) + } +} + +impl From for Error { + fn from(error: InvocationError) -> Self { + Error::Invocation(error) + } +} + +/// Which explicit batch failed. Completed batches are not retried; their +/// decoded values are kept in `completed`. +#[derive(Clone, Debug)] +pub struct BatchError { + /// The failed batch, or `None` when a batch was refused locally before + /// any worker started. + pub index: Option, + pub completed: Vec, + pub error: Error, +} + +impl fmt::Display for BatchError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self.index { + Some(index) => write!(f, "batch {index}: {}", self.error), + None => self.error.fmt(f), + } + } +} + +impl std::error::Error for BatchError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(&self.error) + } +} + +fn lock(mutex: &Mutex) -> MutexGuard<'_, T> { + mutex.lock().unwrap_or_else(|poisoned| poisoned.into_inner()) +} + +/// One discovery: empty until a run succeeds. +type Flight = Arc>>>; + +/// Options for Core discovery and invocation bounds. +#[derive(Clone, Debug, Default)] +pub struct RuntimeOptions { + /// The Core executable; defaults to `$JACQUARD_BIN`, then `jacquard`. + pub binary: Option, + /// Core's prelude directory; defaults to `$JACQUARD_PRELUDE`. + pub prelude: Option, + /// Bounds each invocation's worker lifetime (default 30 seconds). + pub deadline: Option, + /// The session limits; `None` selects the hard limits. + pub limits: Option, +} + +/// Prepares programs against one Core installation. Clones share one +/// version check and prelude discovery. +#[derive(Clone)] +pub struct Runtime { + inner: Arc, +} + +impl fmt::Debug for Runtime { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("Runtime").field("binary", &self.inner.binary).field("prelude", &self.inner.prelude).finish_non_exhaustive() + } +} + +struct RuntimeInner { + binary: String, + prelude: PathBuf, + deadline: Option, + limits: Limits, + /// Version checks and prelude discoveries, each run once per key; a + /// failed run leaves its slot empty for the next caller. + flights: Mutex>, + commands: AtomicUsize, +} + +/// Options for one preparation. +#[derive(Clone, Debug, Default)] +pub struct PrepareOptions { + /// Holds verified prepared programs (default [`DEFAULT_CACHE_DIR`]). + pub cache_dir: Option, +} + +/// Removes a staging directory unless it was published. +struct Staging(PathBuf); + +impl Drop for Staging { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +fn staging_directory(cache: &Path) -> Result { + static NEXT: AtomicUsize = AtomicUsize::new(0); + let nanos = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |elapsed| elapsed.subsec_nanos()); + for _ in 0..100 { + let name = format!("prepare-{}-{nanos:08x}-{}", std::process::id(), NEXT.fetch_add(1, Ordering::SeqCst)); + let path = cache.join(name); + match std::fs::create_dir(&path) { + Ok(()) => return Ok(Staging(path)), + Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => {} + Err(error) => return Err(Error::preparation(format!("cannot create {}: {error}", path.display()))), + } + } + Err(Error::preparation("cannot create a staging directory")) +} + +fn io_error(context: impl fmt::Display) -> impl FnOnce(std::io::Error) -> Error { + move |error| Error::preparation(format!("{context}: {error}")) +} + +/// The prepared metadata in `program.json`. +struct Metadata { + identity: BTreeMap, + names: Vec<(String, Hash)>, + constructors: Vec<(String, Hash)>, + signatures: BTreeMap, + groups: Vec, + files: BTreeMap, +} + +fn string_map(map: &BTreeMap) -> Json { + Json::Object(map.iter().map(|(key, value)| (key.clone(), Json::Str(value.clone()))).collect()) +} + +fn pairs(list: &[(String, Hash)]) -> Json { + Json::Array(list.iter().map(|(name, hash)| Json::Array(vec![name.as_str().into(), hash.as_str().into()])).collect()) +} + +impl Metadata { + fn to_json(&self) -> String { + let groups = self.groups.iter().map(|group| { + Json::Array( + group + .iter() + .map(|entry| { + Json::Object(Object::from([ + ("hash".into(), entry.hash.as_str().into()), + ("name".into(), entry.name.as_str().into()), + ])) + }) + .collect(), + ) + }); + let object = Object::from([ + ("constructors".into(), pairs(&self.constructors)), + ("files".into(), string_map(&self.files)), + ("groups".into(), Json::Array(groups.collect())), + ("identity".into(), string_map(&self.identity)), + ("names".into(), pairs(&self.names)), + ("signatures".into(), string_map(&self.signatures)), + ]); + stringify(&Json::Object(object), 8).expect("bounded metadata") + } + + fn from_json(text: &[u8]) -> Option { + let object = decode_object(text, 8).ok()?; + let map = |key: &str| -> Option> { + object.get(key)?.as_object()?.iter().map(|(key, value)| Some((key.clone(), value.as_str()?.to_owned()))).collect() + }; + let pairs = |key: &str| -> Option> { + object + .get(key)? + .as_array()? + .iter() + .map(|pair| match pair.as_array()? { + [name, hash] => Some((name.as_str()?.to_owned(), Hash::parse(hash.as_str()?).ok()?)), + _ => None, + }) + .collect() + }; + let groups = object + .get("groups")? + .as_array()? + .iter() + .map(|group| { + group + .as_array()? + .iter() + .map(|entry| { + let entry = entry.as_object()?; + Some(Entry { name: entry.get("name")?.as_str()?.to_owned(), hash: Hash::parse(entry.get("hash")?.as_str()?).ok()? }) + }) + .collect() + }) + .collect::>()?; + Some(Metadata { + identity: map("identity")?, + names: pairs("names")?, + constructors: pairs("constructors")?, + signatures: map("signatures")?, + groups, + files: map("files")?, + }) + } +} + +impl Runtime { + /// Validates options; Core does not run yet. + pub fn new(options: RuntimeOptions) -> Result { + // Like the Go SDK, an empty option means unset. + let binary = options + .binary + .filter(|value| !value.is_empty()) + .or_else(|| std::env::var("JACQUARD_BIN").ok().filter(|value| !value.is_empty())); + let prelude = options + .prelude + .filter(|value| !value.as_os_str().is_empty()) + .or_else(|| std::env::var_os("JACQUARD_PRELUDE").filter(|value| !value.is_empty()).map(PathBuf::from)); + let Some(prelude) = prelude else { + return Err(Error::binding( + "set JACQUARD_PRELUDE or RuntimeOptions::prelude; Core and prelude must match the compatibility pin", + )); + }; + let prelude = std::path::absolute(&prelude).map_err(|error| Error::binding(error.to_string()))?; + if prelude.to_str().is_none() { + return Err(Error::binding("the prelude path must be UTF-8")); + } + if options.deadline == Some(Duration::ZERO) { + return Err(Error::binding("deadline must be positive")); + } + let limits = options.limits.unwrap_or(HARD_LIMITS); + limits.validate().map_err(|error| Error::binding(format!("invalid runtime limits: {error}")))?; + Ok(Runtime { + inner: Arc::new(RuntimeInner { + binary: binary.unwrap_or_else(|| "jacquard".into()), + prelude, + deadline: options.deadline, + limits, + flights: Mutex::new(BTreeMap::new()), + commands: AtomicUsize::new(0), + }), + }) + } + + /// Test-only: how many Core commands this runtime has run. + #[doc(hidden)] + pub fn core_commands(&self) -> usize { + self.inner.commands.load(Ordering::SeqCst) + } + + fn command(&self, binary: &Path, arguments: &[&OsStr]) -> Result { + self.inner.commands.fetch_add(1, Ordering::SeqCst); + core::run(binary, &self.inner.prelude, arguments) + } + + /// Runs `work` once per key. Concurrent callers wait for one run; a + /// successful result stays cached, and a failed or panicked run leaves + /// the next caller to try again. + fn once(&self, key: String, work: impl FnOnce() -> Result, Error>) -> Result, Error> { + let slot = lock(&self.inner.flights).entry(key).or_default().clone(); + let mut done = lock(&slot); + if let Some(groups) = done.as_ref() { + return Ok(groups.clone()); + } + let groups = work()?; + *done = Some(groups.clone()); + Ok(groups) + } + + /// Runs `--version` once per executable digest. Like the other SDKs, it + /// accepts output whose last field is the pinned version. + fn check_version(&self, binary: &Path, binary_hash: &str) -> Result<(), Error> { + self.once(format!("version:{binary_hash}"), || { + let version = self.command(binary, &[OsStr::new("--version")])?; + if version.split_whitespace().last() != Some(CORE_VERSION) { + return Err(Error::preparation(format!( + "expected pinned Core version {CORE_VERSION}, found {:?}; the exact release is also required", + version.trim() + ))); + } + Ok(Vec::new()) + }) + .map(drop) + } + + fn prelude_groups(&self, binary: &Path, key: &str) -> Result, Error> { + self.once(format!("prelude:{key}"), || { + let mut groups = Vec::new(); + for name in ["01-prim.jqd", "02-data.jqd", "19-audit.jqd"] { + let path = self.inner.prelude.join(name); + groups.extend(hash_groups(&self.command(binary, &[OsStr::new("hash"), path.as_os_str()])?)?); + } + Ok(groups) + }) + } + + /// Checks, hashes, and installs first-party declaration sources (`.jac` + /// files, or `.jqd` files, in dependency order) into a verified cache + /// keyed by the source bytes, Core executable bytes, prelude contents, + /// and protocol. + pub fn prepare>(&self, sources: &[P], options: &PrepareOptions) -> Result { + let usage = || Error::preparation("pass .jac files or .jqd files in dependency order"); + let extension = |path: &Path| path.extension().and_then(OsStr::to_str).map(|extension| format!(".{extension}")); + let syntax = sources.first().and_then(|source| extension(source.as_ref())).ok_or_else(usage)?; + if !matches!(syntax.as_str(), ".jac" | ".jqd") || sources.iter().any(|source| extension(source.as_ref()).as_ref() != Some(&syntax)) + { + return Err(usage()); + } + let cache = options.cache_dir.clone().unwrap_or_else(|| PathBuf::from(DEFAULT_CACHE_DIR)); + let cache = std::path::absolute(&cache).map_err(io_error("cannot resolve the cache directory"))?; + let binary = core::executable(&self.inner.binary)?; + let binary_hash = core::file_digest(&binary).map_err(io_error(format!("cannot read {}", binary.display())))?; + self.check_version(&binary, &binary_hash)?; + let prelude_files = core::tree(&self.inner.prelude) + .map_err(|error| Error::preparation(format!("cannot read prelude {}: {error}", self.inner.prelude.display())))?; + let prelude_hash = core::map_digest(&prelude_files); + let mut content = Vec::new(); + for (index, source) in sources.iter().enumerate() { + let source = source.as_ref(); + if index > 0 { + content.push(b'\n'); + } + content.extend(std::fs::read(source).map_err(io_error(format!("cannot read {}", source.display())))?); + } + let identity = BTreeMap::from([ + ("binary".to_owned(), binary_hash.clone()), + ("prelude".to_owned(), prelude_hash.clone()), + ("protocol".to_owned(), PROTOCOL.to_owned()), + ("schema".to_owned(), PROGRAM_SCHEMA.to_owned()), + ("source".to_owned(), core::digest(&content)), + ("syntax".to_owned(), syntax.clone()), + ]); + let directory = cache.join(core::map_digest(&identity)); + std::fs::create_dir_all(&cache).map_err(io_error(format!("cannot create {}", cache.display())))?; + if directory.exists() { + return self.load(&directory, &identity, &binary); + } + let staging = staging_directory(&cache)?; + let snapshot = staging.0.join(format!("source{syntax}")); + std::fs::write(&snapshot, &content).map_err(io_error("cannot write the source snapshot"))?; + let output = self.command(&binary, &[OsStr::new("check"), snapshot.as_os_str(), OsStr::new("--print-sigs")])?; + let signatures = output + .split('\n') + .filter_map(|line| line.trim().split_once(" : ")) + .filter(|(name, _)| !name.contains([' ', '\t'])) + .map(|(name, signature)| (name.to_owned(), signature.to_owned())) + .collect(); + let mut groups = self.prelude_groups(&binary, &format!("{binary_hash}{prelude_hash}"))?; + groups.extend(hash_groups(&self.command(&binary, &[OsStr::new("hash"), snapshot.as_os_str()])?)?); + check_groups(&groups)?; + let store = staging.0.join("store"); + self.command(&binary, &[OsStr::new("run"), snapshot.as_os_str(), OsStr::new("--store"), store.as_os_str()])?; + let mut names: Vec<(String, Hash)> = Vec::new(); + let mut constructors: BTreeMap = BTreeMap::new(); + let mut order: Vec = Vec::new(); + let mut seen = BTreeSet::new(); + for entry in groups.iter().flatten() { + if seen.insert(entry.name.clone()) { + names.push((entry.name.clone(), entry.hash.clone())); + } + if constructors.insert(entry.name.clone(), entry.hash.clone()).is_none() { + order.push(entry.name.clone()); + } + } + let constructors = order.into_iter().map(|name| (name.clone(), constructors[&name].clone())).collect(); + let metadata = Metadata { identity: identity.clone(), names, constructors, signatures, groups, files: core::tree(&staging.0)? }; + std::fs::write(staging.0.join("program.json"), metadata.to_json()).map_err(io_error("cannot write program.json"))?; + if let Err(error) = std::fs::rename(&staging.0, &directory) { + // A concurrent preparation published the same artifact first. + if !directory.exists() { + return Err(Error::preparation(format!("cannot publish {}: {error}", directory.display()))); + } + } + drop(staging); + self.load(&directory, &identity, &binary) + } + + fn load(&self, directory: &Path, identity: &BTreeMap, binary: &Path) -> Result { + let fail = |detail: &str| { + Error::preparation(format!( + "prepared cache {} is incomplete or changed ({detail}); remove that cache entry and prepare again", + directory.display() + )) + }; + let text = std::fs::read(directory.join("program.json")).map_err(|error| fail(&error.to_string()))?; + let metadata = Metadata::from_json(&text).ok_or_else(|| fail("unreadable program.json"))?; + let mut observed = core::tree(directory).map_err(|error| fail(&error.to_string()))?; + observed.remove("program.json"); + if &metadata.identity != identity || observed != metadata.files { + return Err(fail("cached artifact digests do not match")); + } + check_groups(&metadata.groups)?; + Program::new(self.clone(), directory.to_owned(), metadata, binary) + } +} + +#[derive(Clone, Debug)] +struct EffectGroup { + identity: Hash, + operations: Vec, +} + +/// A prepared, verified program. +pub struct Program { + runtime: Runtime, + directory: PathBuf, + data: Arc, + host: Host, + interfaces: BTreeMap, + effects: BTreeMap, + services: BTreeMap, +} + +impl fmt::Debug for Program { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("Program").field("directory", &self.directory).field("functions", &self.interfaces.keys()).finish_non_exhaustive() + } +} + +impl Program { + fn new(runtime: Runtime, directory: PathBuf, metadata: Metadata, binary: &Path) -> Result { + let names: BTreeMap = metadata.names.into_iter().collect(); + let constructors = metadata.constructors.into_iter().collect(); + let prelude = runtime.inner.prelude.to_str().expect("checked UTF-8").to_owned(); + let env = std::env::vars_os() + // The carrier takes UTF-8 pairs; other variables are not passed. + .filter_map(|(key, value)| Some((key.into_string().ok()?, value.into_string().ok()?))) + .filter(|(key, _)| key != "JACQUARD_PRELUDE") + .chain([("JACQUARD_PRELUDE".to_owned(), prelude)]) + .collect(); + let store = directory.join("store"); + let carrier = Options { + command: vec![binary.to_str().ok_or_else(|| Error::binding("the Core executable path must be UTF-8"))?.to_owned()], + env: Some(env), + deadline: runtime.inner.deadline, + }; + let host = + Host::new(store.to_string_lossy(), HostOptions { carrier, limits: Some(runtime.inner.limits) }).map_err(Error::binding)?; + let mut interfaces = BTreeMap::new(); + let mut used = BTreeSet::new(); + for (name, signature) in &metadata.signatures { + if let Ok(interface) = parse_signature(signature, &names) { + used.extend(interface.effects.iter().cloned()); + interfaces.insert(name.clone(), interface); + } + } + let mut effects = BTreeMap::new(); + for group in &metadata.groups { + for entry in group.iter().filter(|entry| used.contains(&entry.hash)) { + let operations = group.iter().filter(|other| other.hash != entry.hash).cloned().collect(); + effects.insert(entry.name.clone(), EffectGroup { identity: entry.hash.clone(), operations }); + } + } + let data = Arc::new(Data::new(names, constructors, &metadata.groups, runtime.inner.limits)); + Ok(Program { runtime, directory, data, host, interfaces, effects, services: BTreeMap::new() }) + } + + /// The runtime that prepared this program. + pub fn runtime(&self) -> &Runtime { + &self.runtime + } + + /// The verified cache entry. + pub fn directory(&self) -> &Path { + &self.directory + } + + /// The program's names and identities. + pub fn data(&self) -> &Data { + &self.data + } + + /// Test-only: replaces the worker command for functions bound + /// afterwards. + #[doc(hidden)] + pub fn set_worker_command(&mut self, command: Vec) { + let store = self.host.store.clone(); + let carrier = Options { command, env: None, deadline: self.runtime.inner.deadline }; + self.host = Host::new(store, HostOptions { carrier, limits: Some(self.host.limits) }).expect("valid options"); + } + + /// Configures an effect's service for every function bound afterwards, + /// by effect name such as `"Inventory"`. + pub fn provide(&mut self, effect: &str, operations: Operations) -> Result<&mut Self, Error> { + if !self.effects.contains_key(&normal(effect)) { + return Err(Error::binding(format!("unknown effect {effect}"))); + } + self.services.insert(normal(effect), operations); + Ok(self) + } + + /// Binds a checked public function, using the provided services. + pub fn function(&self, name: &str) -> Result { + self.function_with(name, []) + } + + /// Binds a checked public function with services for its effects, by + /// effect name, overriding [`Program::provide`]. Every operation of each + /// checked root effect must have a handler. + pub fn function_with<'a>(&self, name: &str, effects: impl IntoIterator) -> Result { + let Some(signature) = self.interfaces.get(name) else { + return Err(Error::binding(format!("{name}: absent or unsupported closed v0 signature"))); + }; + let Some(target) = self.data.name(name) else { + return Err(Error::binding(format!("{name}: no public identity"))); + }; + let selected: BTreeMap<&String, &EffectGroup> = + self.effects.iter().filter(|(_, effect)| signature.effects.contains(&effect.identity)).collect(); + if selected.len() != signature.effects.len() { + return Err(Error::binding(format!("{name}: effect metadata is unavailable; use the low-level host module"))); + } + let mut services = self.services.clone(); + for (effect, operations) in effects { + if !selected.contains_key(&normal(effect)) { + return Err(Error::binding(format!("{name} does not use {effect}"))); + } + services.insert(normal(effect), operations); + } + let mut registry = OperationRegistry::new(); + let mut missing = Vec::new(); + for (effect_name, effect) in &selected { + let service = services.get(*effect_name); + // One service may serve several effects, but every name it + // binds must be some effect's operation. + if let Some(key) = service.and_then(|service| { + service + .handlers + .keys() + .find(|key| !self.effects.values().flat_map(|effect| &effect.operations).any(|operation| &operation.name == *key)) + }) { + return Err(Error::binding(format!("{name}: {effect_name} has no operation {key}"))); + } + for operation in &effect.operations { + let Some(handler) = service.and_then(|service| service.handlers.get(&operation.name)).cloned() else { + missing.push(format!("{effect_name}.{}", operation.name)); + continue; + }; + let data = self.data.clone(); + registry + .bind(effect.identity.clone(), operation.hash.clone(), move |context, arguments| handler(&data, context, arguments)) + .map_err(Error::binding)?; + } + } + if !missing.is_empty() { + return Err(Error::binding(format!( + "{name}: missing handlers: {}. Bind every operation in each declared effect before calling", + missing.join(", ") + ))); + } + Ok(Function { + name: name.to_owned(), + signature: signature.clone(), + target: target.clone(), + registry, + host: self.host.clone(), + data: self.data.clone(), + }) + } +} + +/// A function's arguments: a tuple of [`Encode`] values, such as `(35,)`, +/// or a slice of `&dyn Encode`. +pub trait Arguments { + #[doc(hidden)] + fn encode_all(&self, data: &Data, name: &str, first: usize) -> Result, InvalidArgument>; +} + +fn argument(data: &Data, name: &str, index: usize, value: &(impl Encode + ?Sized)) -> Result { + value.encode(&Encoder::root(data, format!("{name}.argument[{index}]"), 3)?) +} + +impl Arguments for [&dyn Encode] { + fn encode_all(&self, data: &Data, name: &str, first: usize) -> Result, InvalidArgument> { + self.iter().enumerate().map(|(index, value)| argument(data, name, first + index, *value)).collect() + } +} + +impl Arguments for [&dyn Encode; N] { + fn encode_all(&self, data: &Data, name: &str, first: usize) -> Result, InvalidArgument> { + self.as_slice().encode_all(data, name, first) + } +} + +impl Arguments for &A { + fn encode_all(&self, data: &Data, name: &str, first: usize) -> Result, InvalidArgument> { + (**self).encode_all(data, name, first) + } +} + +macro_rules! arguments { + ($(($($name:ident $index:tt),*)),*) => {$( + impl<$($name: Encode),*> Arguments for ($($name,)*) { + #[allow(unused_variables)] + fn encode_all(&self, data: &Data, name: &str, first: usize) -> Result, InvalidArgument> { + Ok(vec![$(argument(data, name, first + $index, &self.$index)?),*]) + } + } + )*}; +} + +arguments!((), (A 0), (A 0, B 1), (A 0, B 1, C 2), (A 0, B 1, C 2, D 3), (A 0, B 1, C 2, D 3, E 4), (A 0, B 1, C 2, D 3, E 4, F 5)); + +/// A bound checked function. Clones share their handlers. +#[derive(Clone)] +pub struct Function { + name: String, + signature: Interface, + target: Hash, + registry: OperationRegistry, + host: Host, + data: Arc, +} + +impl fmt::Debug for Function { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("Function").field("name", &self.name).field("target", &self.target).finish_non_exhaustive() + } +} + +impl Function { + /// The public name. + pub fn name(&self) -> &str { + &self.name + } + + /// The checked signature. + pub fn signature(&self) -> &Interface { + &self.signature + } + + /// Validates encoded arguments against the signature and the frame + /// limits, before any worker starts. + fn check(&self, values: Vec) -> Result, Error> { + let expected = self.signature.parameters.len(); + if values.len() != expected { + return Err( + InvalidArgument { message: format!("{}: expected {expected} arguments, received {}", self.name, values.len()) }.into() + ); + } + for (index, (value, parameter)) in values.iter().zip(&self.signature.parameters).enumerate() { + self.data.conform(value, parameter, &format!("{}.argument[{index}]", self.name))?; + } + let limits = &self.host.limits; + invoke_frame(&self.target, &self.signature, &values, &self.registry.entries(), limits) + .and_then(|frame| frame_bytes(&frame, limits.max_frame_bytes, limits.max_json_depth)) + .map_err(|error| InvalidArgument { + message: format!("{}: {error}; reduce input size or use smaller independent batches", self.name), + })?; + Ok(values) + } + + fn arguments(&self, args: &(impl Arguments + ?Sized)) -> Result, Error> { + self.check(args.encode_all(&self.data, &self.name, 0)?) + } + + fn run(&self, values: Vec, cancel: &Cancel) -> Invocation { + Invocation { raw: self.host.invoke(&self.target, &self.signature, &values, &self.registry, cancel), function: self.clone() } + } + + /// Validates the arguments locally, then runs one worker. The error is + /// only a local refusal; Core and carrier failures are in the + /// [`Invocation`]. Nothing is retried. + pub fn invoke(&self, args: impl Arguments) -> Result { + self.invoke_with(args, &Cancel::new()) + } + + /// [`Function::invoke`]; cancelling `cancel` kills the worker. + pub fn invoke_with(&self, args: impl Arguments, cancel: &Cancel) -> Result { + Ok(self.run(self.arguments(&args)?, cancel)) + } + + /// Invokes the function and decodes its result. + pub fn call(&self, args: impl Arguments) -> Result { + self.invoke(args)?.decode() + } + + /// [`Function::call`]; cancelling `cancel` kills the worker. + pub fn call_with(&self, args: impl Arguments, cancel: &Cancel) -> Result { + self.invoke_with(args, cancel)?.decode() + } + + /// Runs a pure function whose final parameter and result are Lists over + /// explicit chunks of `rows`, with `leading` as the other arguments. + /// Every chunk is validated before the first runs. Nothing is retried. + pub fn map_batches( + &self, + leading: impl Arguments, + rows: &[T], + batch_size: usize, + ) -> Result, BatchError> { + self.map_batches_with(leading, rows, batch_size, &Cancel::new()) + } + + /// [`Function::map_batches`]; cancelling `cancel` kills the running + /// worker. + pub fn map_batches_with( + &self, + leading: impl Arguments, + rows: &[T], + batch_size: usize, + cancel: &Cancel, + ) -> Result, BatchError> { + let refuse = |error: Error| BatchError { index: None, completed: Vec::new(), error }; + if batch_size < 1 { + return Err(refuse(InvalidArgument { message: "batch size must be positive".into() }.into())); + } + let list = self.data.name("list"); + let is_list = |t: &BoundaryType| matches!(t, BoundaryType::Nominal(identity, _) if Some(identity) == list); + if !self.signature.effects.is_empty() || !self.signature.parameters.last().is_some_and(is_list) || !is_list(&self.signature.result) + { + return Err(refuse(Error::binding("map_batches needs a pure function with a final List parameter and a List result"))); + } + let leading = leading.encode_all(&self.data, &self.name, 0).map_err(|error| refuse(error.into()))?; + let last = leading.len(); + let batch = |chunk: &[T]| -> Result, Error> { + let mut values = leading.clone(); + values.push(argument(&self.data, &self.name, last, chunk)?); + self.check(values) + }; + batch(&[]).map_err(refuse)?; + let batches = rows.chunks(batch_size).map(batch).collect::, _>>().map_err(refuse)?; + let mut completed = Vec::new(); + for (index, values) in batches.into_iter().enumerate() { + match self.run(values, cancel).decode::>() { + Ok(values) => completed.extend(values), + Err(error) => return Err(BatchError { index: Some(index), completed, error }), + } + } + Ok(completed) + } +} + +/// One completed invocation with its raw evidence. +#[derive(Clone)] +pub struct Invocation { + /// The host's result: Core's frames, the host's frames, and evidence. + pub raw: InvocationResult, + function: Function, +} + +impl fmt::Debug for Invocation { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("Invocation").field("function", &self.function.name).field("raw", &self.raw).finish() + } +} + +impl Invocation { + /// Whether Core reported success. + pub fn ok(&self) -> bool { + self.raw.ok() + } + + /// The primary diagnostic or host code. + pub fn code(&self) -> String { + self.raw.code() + } + + /// A summary for people. + pub fn message(&self) -> String { + self.raw.message() + } + + /// The classified failure, or `None` on success. + pub fn error(&self) -> Option { + (!self.ok()).then(|| InvocationError(Box::new(self.raw.clone()))) + } + + /// Decodes the result. A mapping failure after a successful Core result + /// is an [`Error::Binding`] retaining the raw result. + pub fn decode(&self) -> Result { + if let Some(error) = self.error() { + return Err(error.into()); + } + let Some(value) = self.raw.value.as_ref() else { + return Err(Error::Binding { + message: format!("{}: Core completed without a value", self.function.name), + result: Some(Box::new(self.raw.clone())), + }); + }; + self.function.data.decode(value).map_err(|error| Error::Binding { + message: format!("{}: Core completed, but result mapping failed: {error}", self.function.name), + result: Some(Box::new(self.raw.clone())), + }) + } + + /// Decodes the result into native values. + pub fn value(&self) -> Result { + self.decode() + } + + /// Lists the outcome and each effect request by operation name. + pub fn render(&self) -> String { + let status = if self.ok() { "ok".to_owned() } else { self.message() }; + let mut lines = vec![format!("{}: {status}", self.function.name)]; + for request in &self.raw.requests { + let label = self.function.data.label(&request.operation).map_or_else(|| request.operation.to_string(), str::to_owned); + lines.push(format!(" {}. {label}", request.ordinal)); + } + lines.join("\n") + } + + /// Writes the invocation's frames and host evidence. The record contains + /// argument and result values; it is written only when called. A new + /// file is created owner-only on Unix; an existing file keeps its + /// permissions. + pub fn save(&self, path: impl AsRef) -> std::io::Result<()> { + let raw = &self.raw; + let frames = |list: &[Object]| Json::Array(list.iter().cloned().map(Json::Object).collect()); + let host = Object::from([ + ("evidence".into(), Json::Object(raw.host_evidence.clone())), + ("exit_status".into(), raw.exit_status.map_or(Json::Null, |status| Json::Int(i64::from(status)))), + ("signal".into(), raw.signal.clone().map_or(Json::Null, Json::Str)), + ("stderr_base64".into(), Json::Str(core::base64(&raw.stderr))), + ]); + let record = Object::from([ + ("core_frames".into(), frames(&raw.core_frames)), + ("function".into(), Json::Str(self.function.name.clone())), + ("host".into(), Json::Object(host)), + ("host_frames".into(), frames(&raw.host_frames)), + ("schema".into(), Json::Str("jacquard-host-inspection-v1".into())), + ]); + let text = stringify(&Json::Object(record), 128).map_err(std::io::Error::other)?; + let mut options = std::fs::OpenOptions::new(); + options.write(true).create(true).truncate(true); + #[cfg(unix)] + std::os::unix::fs::OpenOptionsExt::mode(&mut options, 0o600); + std::io::Write::write_all(&mut options.open(path)?, format!("{text}\n").as_bytes()) + } +} diff --git a/rust/src/sdk/operations.rs b/rust/src/sdk/operations.rs new file mode 100644 index 0000000..5cddbd0 --- /dev/null +++ b/rust/src/sdk/operations.rs @@ -0,0 +1,133 @@ +//! Rust closures serving an effect's operations. + +use std::collections::BTreeMap; +use std::marker::PhantomData; +use std::sync::Arc; + +use crate::host::{HandlerContext, HandlerError}; +use crate::protocol::Value; + +use super::data::{Data, Decode, Encode}; +use super::identities::normal; + +/// A typed handler with its argument mapping erased. +pub(crate) type Erased = Arc Result + Send + Sync>; + +/// One effect's service: operation names mapped to Rust closures. +/// +/// Names are store spellings such as `"lookup-stock"`, or Rust spellings +/// such as `"lookup_stock"`. A closure's parameters decode the request's +/// arguments; a leading `&HandlerContext` parameter reports cancellation. +/// It returns `Result`, where `T` encodes the result (`()` +/// for unit). A structured [`HandlerError`] is reported as that failure; any +/// other error raised with `?`, or a panic, is reported as completion +/// unknown. +/// +/// ``` +/// use jacquard_host::host::{HandlerContext, HandlerError}; +/// use jacquard_host::sdk::Operations; +/// +/// let inventory = Operations::new() +/// .on("lookup-stock", |sku: String| Ok(if sku == "sensor" { 20 } else { 0 })) +/// .on("reserve", |context: &HandlerContext, sku: String, units: i64| { +/// if context.is_cancelled() { +/// return Err(HandlerError::refused("cancelled")); +/// } +/// Ok(()) +/// }); +/// ``` +#[derive(Clone, Default)] +pub struct Operations { + pub(crate) handlers: BTreeMap, +} + +impl Operations { + /// An empty service. + pub fn new() -> Self { + Self::default() + } + + /// Serves `operation` with `handler`, replacing any earlier handler for + /// the same operation. + pub fn on>(mut self, operation: &str, handler: F) -> Self { + self.handlers.insert(normal(operation), handler.into_handler()); + self + } +} + +impl std::fmt::Debug for Operations { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_set().entries(self.handlers.keys()).finish() + } +} + +/// A closure that can serve an operation; see [`Operations`]. `M` describes +/// its parameters. +pub trait IntoHandler: Send + Sync + 'static { + #[doc(hidden)] + fn into_handler(self) -> Erased; +} + +/// The parameter description of a handler whose first parameter is +/// `&HandlerContext`. +pub struct WithContext(PhantomData); + +fn arguments_failed() -> HandlerError { + HandlerError::outside_failed("handler argument cannot be mapped") +} + +fn result(data: &Data, outcome: Result) -> Result { + data.encode(&outcome?).map_err(|_| HandlerError::outside_failed("handler result cannot be represented")) +} + +macro_rules! handlers { + ($($name:ident),*) => { + impl IntoHandler<($($name,)*)> for F + where + F: Fn($($name),*) -> Result + Send + Sync + 'static, + T: Encode, + $($name: Decode,)* + { + #[allow(non_snake_case, unused_variables, unused_mut)] + fn into_handler(self) -> Erased { + Arc::new(move |data, _, arguments| { + const COUNT: usize = <[&str]>::len(&[$(stringify!($name)),*]); + if arguments.len() != COUNT { + return Err(HandlerError::outside_failed("handler parameter count differs from the request")); + } + let mut arguments = arguments.iter(); + $(let $name: $name = data.decode(arguments.next().expect("counted")).map_err(|_| arguments_failed())?;)* + result(data, self($($name),*)) + }) + } + } + + impl IntoHandler> for F + where + F: Fn(&HandlerContext, $($name),*) -> Result + Send + Sync + 'static, + T: Encode, + $($name: Decode,)* + { + #[allow(non_snake_case, unused_variables, unused_mut)] + fn into_handler(self) -> Erased { + Arc::new(move |data, context, arguments| { + const COUNT: usize = <[&str]>::len(&[$(stringify!($name)),*]); + if arguments.len() != COUNT { + return Err(HandlerError::outside_failed("handler parameter count differs from the request")); + } + let mut arguments = arguments.iter(); + $(let $name: $name = data.decode(arguments.next().expect("counted")).map_err(|_| arguments_failed())?;)* + result(data, self(context, $($name),*)) + }) + } + } + }; +} + +handlers!(); +handlers!(A); +handlers!(A, B); +handlers!(A, B, C); +handlers!(A, B, C, D); +handlers!(A, B, C, D, E); +handlers!(A, B, C, D, E, G); diff --git a/rust/src/sdk/tests.rs b/rust/src/sdk/tests.rs new file mode 100644 index 0000000..116d4d1 --- /dev/null +++ b/rust/src/sdk/tests.rs @@ -0,0 +1,434 @@ +//! Offline SDK tests with synthetic identities and a fake Core. + +use std::collections::BTreeMap; + +use crate::host::{HandlerContext, HandlerError}; +use crate::protocol::{BoundaryType, FailureCategory, HARD_LIMITS, Hash, Limits, Value}; +use crate::{record, variants}; + +use super::data::Data; +use super::identities::{Entry, check_groups, display_to_store, hash_groups, normal, parse_signature}; +use super::{Dynamic, Error, Operations, PrepareOptions, Runtime, RuntimeOptions}; + +fn id(letter: char) -> Hash { + Hash::parse(&letter.to_string().repeat(64)).unwrap() +} + +fn numbered(n: u32) -> Hash { + Hash::parse(&format!("{n:064x}")).unwrap() +} + +fn nominal(identity: Hash, arguments: Vec) -> BoundaryType { + BoundaryType::Nominal(identity, arguments) +} + +/// Data with synthetic identities for the builtins, an Incident record, an +/// Urgency enum, and a Triage union. +fn fixture(limits: Limits) -> Data { + let names: BTreeMap = [ + ("int", id('1')), + ("real", id('2')), + ("text", id('3')), + ("bool", id('4')), + ("hash", id('5')), + ("list", id('6')), + ("option", id('7')), + ("incident", id('8')), + ("urgency", id('9')), + ("triage", numbered(10)), + ] + .into_iter() + .map(|(name, hash)| (name.to_owned(), hash)) + .collect(); + let constructors: BTreeMap = [ + ("true", id('a')), + ("false", id('b')), + ("some", id('c')), + ("none", id('d')), + ("cons", id('e')), + ("nil", id('f')), + ("incident", id('0')), + ("page-now", numbered(1)), + ("watch-only", numbered(2)), + ("routed", numbered(3)), + ("invalid-incident", numbered(4)), + ] + .into_iter() + .map(|(name, hash)| (name.to_owned(), hash)) + .collect(); + let group = |head: &str, members: &[&str]| { + let mut group = vec![Entry { name: head.into(), hash: names[head].clone() }]; + group.extend(members.iter().map(|name| Entry { name: (*name).into(), hash: constructors[*name].clone() })); + group + }; + let groups = + [group("incident", &["incident"]), group("urgency", &["page-now", "watch-only"]), group("triage", &["routed", "invalid-incident"])]; + Data::new(names, constructors, &groups, limits) +} + +#[derive(Clone, Debug, PartialEq)] +struct Incident { + id: String, + severity: i64, + owner: Option, +} +record!(Incident = "Incident" { id, severity, owner }); + +#[derive(Clone, Copy, Debug, PartialEq)] +enum Urgency { + Page, + Watch, +} +variants!(Urgency { Page = "PageNow", Watch = "WatchOnly" }); + +#[derive(Debug, PartialEq)] +enum Triage { + Routed { id: String, urgency: Urgency }, + InvalidIncident(String), +} +variants!(Triage { Routed { id, urgency } = "Routed", InvalidIncident(id) = "InvalidIncident" }); + +#[test] +fn signatures_and_groups() { + let names: BTreeMap = + [("list", '6'), ("line", '8'), ("int", '1'), ("option", '7'), ("text", '3'), ("inventory", '9'), ("dispatch", 'a')] + .into_iter() + .map(|(name, letter)| (name.to_owned(), id(letter))) + .collect(); + let spec = parse_signature("(List Line, Int) ->{Inventory, Dispatch} Option Text", &names).unwrap(); + assert_eq!(spec.parameters, [nominal(id('6'), vec![nominal(id('8'), vec![])]), nominal(id('1'), vec![])]); + assert_eq!(spec.effects, [id('9'), id('a')]); + assert_eq!(spec.result, nominal(id('7'), vec![nominal(id('3'), vec![])])); + assert_eq!(parse_signature("() ->{} ()", &names).unwrap().result, BoundaryType::Tuple(vec![])); + assert_eq!(parse_signature("((Int, Text)) ->{} (Int)", &names).unwrap().result, nominal(id('1'), vec![])); + assert_eq!( + parse_signature("((Int, Text)) ->{} Int", &names).unwrap().parameters, + [BoundaryType::Tuple(vec![nominal(id('1'), vec![]), nominal(id('3'), vec![])])] + ); + for text in ["forall a | e. (List a) ->{| e} Bool", "(Missing) ->{} Int", "(Int) ->{} Int extra)", "(Int) ->{} Int $"] { + assert!(parse_signature(text, &names).is_err(), "accepted {text:?}"); + } + let output = format!("0 {}\n0:line {}\n0:line {}\nnoise\n1:other {}\n", id('1'), id('1'), id('2'), id('3')); + let groups = hash_groups(&output).unwrap(); + assert_eq!(groups.len(), 2); + assert_eq!(groups[0][1].hash, id('2')); + assert!(matches!(hash_groups("nothing here"), Err(Error::Binding { .. }))); + let entry = |name: &str, hash| Entry { name: name.into(), hash }; + assert!(check_groups(&[vec![entry("x", id('1'))], vec![entry("x", id('2'))]]).is_err()); + assert!(check_groups(&[vec![entry("x", id('1'))], vec![entry("x", id('1'))]]).is_ok()); + for (input, output) in [("WorkToday", "work-today"), ("PageNow", "page-now"), ("text", "text"), ("Inventory", "inventory")] { + assert_eq!(display_to_store(input), output); + } + assert_eq!(normal("lookup_stock"), "lookup-stock"); +} + +#[test] +fn native_values_roundtrip_and_conform_to_checked_types() { + let data = fixture(HARD_LIMITS); + let rows = + vec![Incident { id: "a".into(), severity: 5, owner: None }, Incident { id: "b".into(), severity: 2, owner: Some("ops".into()) }]; + let list = nominal(id('6'), vec![nominal(id('8'), vec![])]); + let value = data.encode(&rows).unwrap(); + data.conform(&value, &list, "rows").unwrap(); + assert_eq!(data.decode::>(&value).unwrap(), rows); + let real = nominal(id('2'), vec![]); + for number in [1.0, -0.0, f64::INFINITY, f64::NAN] { + let value = data.encode(&number).unwrap(); + data.conform(&value, &real, "real").unwrap(); + assert_eq!(data.decode::(&value).unwrap().to_bits(), number.to_bits()); + } + let integer = nominal(id('1'), vec![]); + assert!(matches!(data.encode(&(1u64 << 63)), Err(Error::InvalidArgument(_)))); + assert!(matches!(data.encode(&i64::MIN), Err(Error::InvalidArgument(_)))); + assert_eq!(data.encode(&crate::json::INT_MAX).unwrap(), Value::Int(crate::json::INT_MAX)); + assert!(data.decode::(&Value::Int(300)).is_err()); + assert!(data.decode::(&Value::Int(-1)).is_err()); + let refused = |value: Value, expected: &BoundaryType| data.conform(&value, expected, "argument[0]").unwrap_err().message; + assert_eq!(refused(data.encode("x").unwrap(), &integer), "argument[0]: expected an integer"); + assert_eq!(refused(data.encode(&3).unwrap(), &real), "argument[0]: expected a Real (f64)"); + assert!(refused(data.encode(&Urgency::Page).unwrap(), &nominal(id('8'), vec![])).contains("expected a incident value")); + let late = vec!["ok".to_owned(), "ok".to_owned()]; + let texts = data.encode(&late).unwrap(); + assert_eq!(refused(texts, &nominal(id('6'), vec![integer.clone()])), "argument[0][0]: expected an integer"); + let unit = BoundaryType::Tuple(vec![]); + data.conform(&data.encode(&()).unwrap(), &unit, "unit").unwrap(); + assert!(data.conform(&data.encode(&(1, 2)).unwrap(), &unit, "unit").is_err()); + // A record with an unknown constructor is refused at its location. + struct Unknown { + field: i64, + } + record!(Unknown = "Unknown" { field }); + let error = data.encode(&Unknown { field: 1 }).unwrap_err().to_string(); + assert_eq!(error, "value: unknown constructor Unknown"); + // A record over a builtin constructor is refused, as Dynamic is. + struct Wrapped { + inner: i64, + } + record!(Wrapped = "Some" { inner }); + assert!(data.encode(&Wrapped { inner: 1 }).unwrap_err().to_string().contains("native form of builtin constructor Some")); +} + +#[test] +fn options_enums_unions_and_dynamic_values() { + let data = fixture(HARD_LIMITS); + let routed = Triage::Routed { id: "r".into(), urgency: Urgency::Page }; + let value = data.encode(&routed).unwrap(); + assert_eq!(value, Value::Constructor(numbered(3), vec![Value::Text("r".into()), Value::Constructor(numbered(1), vec![])])); + assert_eq!(data.decode::(&value).unwrap(), routed); + let invalid = data.encode(&Triage::InvalidIncident("x".into())).unwrap(); + assert_eq!(data.decode::(&invalid).unwrap(), Triage::InvalidIncident("x".into())); + assert!(data.decode::(&invalid).unwrap_err().to_string().contains("unknown Urgency case")); + let short = Value::Constructor(numbered(3), vec![Value::Text("r".into())]); + assert!(data.decode::(&short).unwrap_err().to_string().contains("wrong constructor or field count")); + let option = nominal(id('7'), vec![nominal(id('7'), vec![nominal(id('1'), vec![])])]); + for nested in [None, Some(None), Some(Some(3i64))] { + let value = data.encode(&nested).unwrap(); + data.conform(&value, &option, "option").unwrap(); + assert_eq!(data.decode::>>(&value).unwrap(), nested); + let expected = match nested { + None => Dynamic::Option(None), + Some(None) => Dynamic::Option(Some(Box::new(Dynamic::Option(None)))), + Some(Some(n)) => Dynamic::Option(Some(Box::new(Dynamic::Option(Some(Box::new(Dynamic::Int(n))))))), + }; + assert_eq!(data.decode::(&value).unwrap(), expected); + } + let watch = data.make("WatchOnly", []).unwrap(); + assert_eq!(watch, Dynamic::variant("watch-only", [])); + assert_eq!(data.decode::(&data.encode(&watch).unwrap()).unwrap(), Urgency::Watch); + assert!(matches!(data.make("Missing", []), Err(Error::Binding { .. }))); + assert!(data.encode(&Dynamic::variant("Some", [1.into()])).is_err(), "a builtin constructor by name"); + let dynamic = Dynamic::List(vec![Dynamic::Tuple(vec![true.into(), "x".into(), 2.5.into()]), Dynamic::Hash(id('5'))]); + assert_eq!(data.decode::(&data.encode(&dynamic).unwrap()).unwrap(), dynamic); + assert!(data.decode::(&Value::Constructor(id('9'), vec![])).unwrap_err().to_string().contains("unmapped constructor")); + assert!(data.encode(&vec![(); 40]).unwrap_err().to_string().contains("JSON depth"), "a list deeper than the JSON depth"); + assert_eq!(data.identity("Urgency").unwrap(), id('9')); + assert!(data.identity("missing").is_err()); +} + +#[test] +fn selected_limits_bound_mapped_values() { + let limits = Limits { max_collection_items: 2, max_text_bytes: 3, ..HARD_LIMITS }; + let data = fixture(limits); + assert!(data.encode(&vec![1, 2, 3]).unwrap_err().to_string().contains("max_collection_items")); + assert!(data.encode("four").unwrap_err().to_string().contains("max_text_bytes")); + let long = HARD_LIMITS_DATA.with(|data| data.encode(&vec![1, 2, 3]).unwrap()); + assert!(data.decode::>(&long).unwrap_err().to_string().contains("selected limit")); + let nested = Incident { id: "abcd".into(), severity: 1, owner: None }; + assert_eq!(data.encode(&nested).unwrap_err().to_string(), "value.id: text exceeds max_text_bytes"); +} + +thread_local! { + static HARD_LIMITS_DATA: Data = fixture(HARD_LIMITS); +} + +fn failure(error: HandlerError) -> Option { + match error { + HandlerError::Failure(category, _) => Some(category), + HandlerError::Unknown => None, + } +} + +#[test] +fn arguments_are_tuples_or_dynamic_slices() { + use super::{Arguments, Encode}; + let data = fixture(HARD_LIMITS); + let tuple = (35, "x", Some(true)).encode_all(&data, "f", 0).unwrap(); + let values: [&dyn Encode; 3] = [&35, &"x", &Some(true)]; + assert_eq!(values.encode_all(&data, "f", 0).unwrap(), tuple); + assert_eq!(values[..].encode_all(&data, "f", 0).unwrap(), tuple); + assert!(().encode_all(&data, "f", 0).unwrap().is_empty()); + let error = (1, u64::MAX).encode_all(&data, "f", 2).unwrap_err(); + assert_eq!(error.message, "f.argument[3]: integer is outside Core's 63-bit Int range"); +} + +#[test] +fn closures_adapt_to_handlers() { + let data = fixture(HARD_LIMITS); + let context = HandlerContext::for_test(); + let calls = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let counted = calls.clone(); + let service = Operations::new() + .on("read-document", move |context: &HandlerContext, key: String| { + counted.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + match key.as_str() { + _ if context.is_cancelled() => Err(HandlerError::completion_unknown("cancelled")), + "example" => Ok("text".to_owned()), + _ => Err(HandlerError::refused("unknown document")), + } + }) + .on("forget", |_: String| -> Result<(), HandlerError> { Err(std::io::Error::other("private detail"))? }) + .on("lookup_stock", |sku: String| Ok(sku.len())) + .on("pair", || Ok((3, 4))) + .on("route", |incident: Incident| Ok(Triage::Routed { id: incident.id, urgency: Urgency::Watch })) + .on("huge", || Ok(u64::MAX)); + let handler = |name: &str| service.handlers[name].clone(); + let text = |text: &str| Value::Text(text.into()); + assert_eq!(handler("read-document")(&data, &context, &[text("example")]).unwrap(), text("text")); + assert_eq!(calls.load(std::sync::atomic::Ordering::SeqCst), 1); + let refusal = handler("read-document")(&data, &context, &[text("other")]).unwrap_err(); + assert_eq!(failure(refusal), Some(FailureCategory::RefusedAuthority)); + let arity = handler("read-document")(&data, &context, &[]).unwrap_err(); + assert_eq!(arity, HandlerError::outside_failed("handler parameter count differs from the request")); + let mapping = handler("read-document")(&data, &context, &[Value::Int(1)]).unwrap_err(); + assert_eq!(mapping, HandlerError::outside_failed("handler argument cannot be mapped")); + assert_eq!(calls.load(std::sync::atomic::Ordering::SeqCst), 2, "a handler ran with unmappable arguments"); + assert_eq!(handler("forget")(&data, &context, &[text("x")]).unwrap_err(), HandlerError::Unknown, "a plain error stays unclassified"); + assert_eq!(handler("lookup-stock")(&data, &context, &[text("abc")]).unwrap(), Value::Int(3)); + assert_eq!(handler("pair")(&data, &context, &[]).unwrap(), Value::Tuple(vec![Value::Int(3), Value::Int(4)])); + let incident = data.encode(&Incident { id: "i".into(), severity: 1, owner: None }).unwrap(); + assert_eq!( + data.decode::(&handler("route")(&data, &context, &[incident]).unwrap()).unwrap(), + Triage::Routed { id: "i".into(), urgency: Urgency::Watch } + ); + let huge = handler("huge")(&data, &context, &[]).unwrap_err(); + assert_eq!(huge, HandlerError::outside_failed("handler result cannot be represented")); + let replaced = Operations::new().on("op", || Ok(1)).on("op", || Ok(2)); + assert_eq!(replaced.handlers["op"](&data, &context, &[]).unwrap(), Value::Int(2)); +} + +#[test] +fn runtime_options_are_validated_before_core_runs() { + let prelude = Some(std::env::temp_dir()); + let deadline = + Runtime::new(RuntimeOptions { prelude: prelude.clone(), deadline: Some(std::time::Duration::ZERO), ..Default::default() }); + assert!(matches!(deadline, Err(Error::Binding { .. }))); + let limits = Limits { max_frame_bytes: 0, ..HARD_LIMITS }; + let invalid = Runtime::new(RuntimeOptions { prelude: prelude.clone(), limits: Some(limits), ..Default::default() }); + assert!(matches!(invalid, Err(Error::Binding { message, .. }) if message.starts_with("invalid runtime limits"))); + let runtime = Runtime::new(RuntimeOptions { prelude, binary: Some("/no/such/jacquard".into()), ..Default::default() }).unwrap(); + for sources in [vec!["rules.txt"], vec![], vec!["a.jac", "b.jqd"]] { + assert!(matches!(runtime.prepare(&sources, &PrepareOptions::default()), Err(Error::Preparation(_))), "{sources:?}"); + } + let missing = runtime.prepare(&["rules.jac"], &PrepareOptions::default()).err().unwrap().to_string(); + assert!(missing.contains("cannot find Core executable /no/such/jacquard"), "{missing}"); + assert_eq!(runtime.core_commands(), 0); +} + +/// A fake Core: a shell script that logs its arguments, answers +/// `--version`, leaves a descendant holding its output, and fails `check`. +#[cfg(unix)] +mod fake { + use std::os::unix::fs::PermissionsExt; + use std::path::PathBuf; + use std::time::{Duration, Instant}; + + use super::super::{Error, PrepareOptions, Runtime, RuntimeOptions}; + + pub struct Fake { + pub root: PathBuf, + pub runtime: Runtime, + } + + impl Drop for Fake { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.root); + } + } + + impl Fake { + pub fn new(name: &str, version: &str) -> Fake { + Fake::with(name, version, "check) echo 'boom' >&2; exit 3 ;;") + } + + /// A fake whose `check` and `hash` succeed and whose `run` fails. + pub fn hashing(name: &str) -> Fake { + let hash = "0".repeat(64); + Fake::with(name, crate::compat::CORE_VERSION, &format!("check) ;;\nhash) echo '0:x {hash}' ;;\nrun) exit 4 ;;")) + } + + fn with(name: &str, version: &str, cases: &str) -> Fake { + let root = std::env::temp_dir().join(format!("jacquard-host-rust-fake-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + std::fs::create_dir_all(root.join("prelude")).unwrap(); + std::fs::write(root.join("prelude/01-prim.jqd"), "prelude\n").unwrap(); + std::fs::write(root.join("rules.jac"), "rules\n").unwrap(); + let log = root.join("log"); + let script = format!( + "#!/bin/sh\necho \"$1\" >> '{}'\ncase \"$1\" in\n--version) echo 'jacquard {version}'; sleep 30 & ;;\n{cases}\nesac\n", + log.display() + ); + let binary = root.join("jacquard"); + std::fs::write(&binary, script).unwrap(); + std::fs::set_permissions(&binary, std::fs::Permissions::from_mode(0o755)).unwrap(); + let options = + RuntimeOptions { binary: Some(binary.to_string_lossy().into()), prelude: Some(root.join("prelude")), ..Default::default() }; + Fake { runtime: Runtime::new(options).unwrap(), root } + } + + pub fn prepare(&self) -> (Error, Duration) { + self.prepare_source("rules.jac") + } + + pub fn prepare_source(&self, name: &str) -> (Error, Duration) { + let source = self.root.join(name); + if !source.exists() { + std::fs::write(&source, format!("{name}\n")).unwrap(); + } + let started = Instant::now(); + let options = PrepareOptions { cache_dir: Some(self.root.join("cache")) }; + let error = self.runtime.prepare(&[source], &options).expect_err("the fake Core fails"); + (error, started.elapsed()) + } + + pub fn calls(&self, command: &str) -> usize { + std::fs::read_to_string(self.root.join("log")).unwrap_or_default().lines().filter(|line| *line == command).count() + } + + pub fn staging_left(&self) -> bool { + let entries = std::fs::read_dir(self.root.join("cache")); + entries.is_ok_and(|entries| entries.flatten().any(|entry| entry.file_name().to_string_lossy().starts_with("prepare-"))) + } + } +} + +#[cfg(unix)] +#[test] +fn version_is_checked_once_and_failures_are_reported_without_waiting_for_descendants() { + let fake = fake::Fake::new("once", crate::compat::CORE_VERSION); + let errors: Vec<_> = std::thread::scope(|scope| { + let workers: Vec<_> = (0..4).map(|_| scope.spawn(|| fake.prepare())).collect(); + workers.into_iter().map(|worker| worker.join().unwrap()).collect() + }); + for (error, elapsed) in &errors { + assert!(matches!(error, Error::Preparation(message) if message == "Core check failed: boom"), "{error}"); + assert!(elapsed.as_secs() < 10, "a descendant holding Core's output delayed preparation: {elapsed:?}"); + } + assert_eq!(fake.calls("--version"), 1); + assert_eq!(fake.calls("check"), 4); + assert!(!fake.staging_left(), "a staging directory was left behind"); +} + +#[cfg(unix)] +#[test] +fn a_version_mismatch_is_refused_and_checked_again() { + let fake = fake::Fake::new("mismatch", "9.9.9"); + for _ in 0..2 { + let (error, _) = fake.prepare(); + let message = error.to_string(); + assert!(message.contains("expected pinned Core version") && message.contains("jacquard 9.9.9"), "{message}"); + } + assert_eq!(fake.calls("--version"), 2); + assert_eq!(fake.calls("check"), 0); +} + +#[cfg(unix)] +#[test] +fn prelude_discovery_runs_once_per_core_and_prelude() { + let fake = fake::Fake::hashing("prelude"); + for source in ["rules.jac", "other.jac"] { + let (error, _) = fake.prepare_source(source); + assert!(error.to_string().starts_with("Core run failed"), "{error}"); + } + // Three prelude files once, then one local hash per source. + assert_eq!(fake.calls("hash"), 5); + assert_eq!(fake.calls("--version"), 1); + assert_eq!(fake.calls("run"), 2); +} + +#[test] +fn empty_runtime_options_mean_unset() { + let options = RuntimeOptions { binary: Some(String::new()), prelude: Some(std::env::temp_dir()), ..Default::default() }; + let runtime = Runtime::new(options).unwrap(); + let expected = std::env::var("JACQUARD_BIN").ok().filter(|value| !value.is_empty()).unwrap_or_else(|| "jacquard".into()); + assert!(format!("{runtime:?}").contains(&format!("{expected:?}")), "{runtime:?}"); +} diff --git a/rust/tests/sdk_descriptors.rs b/rust/tests/sdk_descriptors.rs new file mode 100644 index 0000000..256402a --- /dev/null +++ b/rust/tests/sdk_descriptors.rs @@ -0,0 +1,69 @@ +//! A Core command whose descendant leaves its process group and keeps the +//! output open fails preparation after the grace period, without leaking +//! reader threads or pipe descriptors. Its own binary, so no other test +//! changes the counts. + +#![cfg(target_os = "linux")] + +use std::os::unix::fs::PermissionsExt; +use std::time::{Duration, Instant}; + +use jacquard_host::sdk::{Error, PrepareOptions, Runtime, RuntimeOptions}; + +fn count(directory: &str) -> usize { + std::fs::read_dir(directory).unwrap().count() +} + +#[test] +fn an_escaped_descendant_holding_core_output_leaks_nothing() { + let root = std::env::temp_dir().join(format!("jacquard-host-rust-escaped-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + std::fs::create_dir_all(root.join("prelude")).unwrap(); + std::fs::write(root.join("prelude/01-prim.jqd"), "prelude\n").unwrap(); + std::fs::write(root.join("rules.jac"), "rules\n").unwrap(); + let pids = root.join("pids"); + // setsid takes the sleeper out of the command's process group, so the + // group kill cannot reach it; it inherits stdout and stderr. + let script = format!( + "#!/bin/sh\ncase \"$1\" in\n--version) echo 'jacquard {}'; setsid sleep 30 & echo $! >> '{}' ;;\nesac\n", + jacquard_host::compat::CORE_VERSION, + pids.display() + ); + let binary = root.join("jacquard"); + std::fs::write(&binary, script).unwrap(); + std::fs::set_permissions(&binary, std::fs::Permissions::from_mode(0o755)).unwrap(); + let options = PrepareOptions { cache_dir: Some(root.join("cache")) }; + let attempt = || { + let runtime = Runtime::new(RuntimeOptions { + binary: Some(binary.to_string_lossy().into()), + prelude: Some(root.join("prelude")), + ..Default::default() + }) + .unwrap(); + let started = Instant::now(); + let error = runtime.prepare(&[root.join("rules.jac")], &options).expect_err("output stays open"); + (error, started.elapsed()) + }; + // Kills the escaped sleepers and removes the scratch directory even when + // an assertion fails. + struct Cleanup(std::path::PathBuf); + impl Drop for Cleanup { + fn drop(&mut self) { + for pid in std::fs::read_to_string(self.0.join("pids")).unwrap_or_default().split_whitespace() { + let _ = std::process::Command::new("kill").arg("-9").arg(pid).status(); + } + let _ = std::fs::remove_dir_all(&self.0); + } + } + let _cleanup = Cleanup(root.clone()); + // Warm up thread-local and allocator state before counting. + let _ = attempt(); + let (threads, descriptors) = (count("/proc/self/task"), count("/proc/self/fd")); + for _ in 0..3 { + let (error, elapsed) = attempt(); + assert!(matches!(&error, Error::Preparation(message) if message.contains("its output did not close")), "{error}"); + assert!(elapsed < Duration::from_millis(1800), "the grace period took {elapsed:?}"); + } + let after = (count("/proc/self/task"), count("/proc/self/fd")); + assert_eq!(after, (threads, descriptors), "threads and descriptors grew"); +} diff --git a/rust/tests/sdk_examples.rs b/rust/tests/sdk_examples.rs new file mode 100644 index 0000000..c86daf1 --- /dev/null +++ b/rust/tests/sdk_examples.rs @@ -0,0 +1,143 @@ +//! The example projects run as commands through the Rust SDK against the +//! real Core named by `JACQUARD_BIN` and `JACQUARD_PRELUDE`; each test skips +//! without them. + +#[allow(dead_code)] +#[path = "../examples/jacquard-examples.rs"] +mod examples; + +use std::path::PathBuf; +use std::sync::atomic::{AtomicUsize, Ordering}; + +use examples::Failure; +use jacquard_host::host::Failure as InvocationFailure; +use jacquard_host::json::{Json, decode}; + +static NEXT: AtomicUsize = AtomicUsize::new(0); + +/// A scratch directory, removed on drop. +struct Scratch(PathBuf); + +impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +fn scratch() -> Option { + if std::env::var_os("JACQUARD_BIN").is_none() || std::env::var_os("JACQUARD_PRELUDE").is_none() { + assert!(std::env::var_os("JACQUARD_REQUIRE_LIVE").is_none(), "live tests need JACQUARD_BIN and JACQUARD_PRELUDE"); + eprintln!("skipped: live tests need JACQUARD_BIN and JACQUARD_PRELUDE"); + return None; + } + let root = + std::env::temp_dir().join(format!("jacquard-host-rust-examples-{}-{}", std::process::id(), NEXT.fetch_add(1, Ordering::SeqCst))); + std::fs::create_dir_all(&root).unwrap(); + Some(Scratch(root)) +} + +fn command(arguments: &[&str]) -> (String, Result<(), Failure>) { + let mut out = Vec::new(); + let arguments: Vec = arguments.iter().map(|argument| argument.to_string()).collect(); + let result = examples::run(&arguments, &mut out); + (String::from_utf8(out).unwrap(), result) +} + +#[test] +fn live_all_four_projects_run_as_commands() { + let Some(root) = scratch() else { return }; + for project in examples::PROJECTS { + let workdir = root.0.join(project); + let workdir = workdir.to_str().unwrap(); + let (output, result) = command(&["--workdir", workdir, project]); + assert!(result.is_ok(), "{project}: {result:?}\n{output}"); + match project { + "quickstart" => assert_eq!(output, "Order total 35: shipping 5\nOrder total 75: shipping 0\n"), + "planner" => assert!(output.contains("\"score\": 16") && output.contains("planner.plan: ok"), "{output}"), + "triage" => { + let rows = decode(output.trim().as_bytes(), 16).unwrap(); + let urgencies: Vec<&str> = + rows.as_array().unwrap().iter().map(|row| row.as_object().unwrap()["urgency"].as_str().unwrap()).collect(); + assert_eq!(urgencies, ["PAGE", "TODAY", "WATCH"], "{output}"); + } + _ => { + assert!(output.contains("\"$tag\": \"Fulfilled\""), "{output}"); + let (repeated, result) = command(&["--workdir", workdir, project]); + assert!(result.is_ok() && repeated.contains("DuplicateOrder") && repeated.contains("\"deliveries\":1"), "{repeated}"); + } + } + } + let recovery = root.0.join("recovery"); + let recovery = recovery.to_str().unwrap(); + let (output, result) = command(&["--workdir", recovery, "--fault", "publish-failed", "fulfillment"]); + match result { + Err(Failure::Sdk(error)) => { + assert_eq!(error.failure(), Some(InvocationFailure::OperationFailed)); + assert_eq!(error.result().unwrap().code(), "E1613"); + } + other => panic!("publish failure: {other:?}"), + } + assert!(output.contains("\"pendingDeliveries\":1"), "{output}"); + // A failed explicit delivery prints no count. + let (output, result) = command(&["--workdir", recovery, "--fault", "publish-failed", "--deliver-pending", "fulfillment"]); + assert!(matches!(result, Err(Failure::Other(_))) && !output.contains("delivered"), "{output}"); + for expected in ["{\"delivered\":1}", "{\"delivered\":0}"] { + let (output, result) = command(&["--workdir", recovery, "--deliver-pending", "fulfillment"]); + assert!(result.is_ok() && output.contains(expected), "deliver pending: {result:?} {output}"); + } + let inspect = root.0.join("triage-inspect"); + let (_, result) = command(&["--workdir", root.0.join("triage").to_str().unwrap(), "--inspect", inspect.to_str().unwrap(), "triage"]); + assert!(result.is_ok(), "{result:?}"); + assert!(PathBuf::from(format!("{}.0.json", inspect.display())).exists(), "inspection not saved"); + assert!(matches!(command(&["unknown"]).1, Err(Failure::Other(_)))); + assert!(matches!(command(&["--fault", "nope", "fulfillment"]).1, Err(Failure::Other(_)))); +} + +#[test] +fn live_partial_completion_faults_recover_explicitly_across_restart() { + let Some(root) = scratch() else { return }; + for (fault, code, deliveries) in [("after-commit", "E1612", 1), ("publish-failed", "E1613", 1), ("after-delivery", "E1612", 0)] { + let workdir = root.0.join(fault); + let workdir = workdir.to_str().unwrap(); + let (output, result) = command(&["--workdir", workdir, "--fault", fault, "fulfillment"]); + let Err(Failure::Sdk(error)) = result else { panic!("{fault}: {result:?}") }; + assert_eq!(error.result().map(|result| result.code()), Some(code.to_owned()), "{fault}: {error}"); + assert!(output.contains("\"bookings\":1"), "{fault}: {output}"); + // A restart sees the committed booking; recovery is explicit. + let (output, result) = command(&["--workdir", workdir, "--deliver-pending", "fulfillment"]); + assert!(result.is_ok() && output.contains(&format!("{{\"delivered\":{deliveries}}}")), "{fault}: {result:?} {output}"); + let (output, result) = command(&["--workdir", workdir, "fulfillment"]); + assert!(result.is_ok() && output.contains("DuplicateOrder") && output.contains("\"pendingDeliveries\":0"), "{fault}: {output}"); + let summary = output.trim().lines().last().unwrap(); + assert_eq!(decode(summary.as_bytes(), 4).unwrap().as_object().unwrap()["deliveries"], Json::Int(1), "{fault}"); + } +} + +/// The example binary `cargo test` builds beside this test's binary. +fn example_binary() -> PathBuf { + let deps = std::env::current_exe().unwrap().parent().unwrap().to_owned(); + deps.parent().unwrap().join("examples").join(format!("jacquard-examples{}", std::env::consts::EXE_SUFFIX)) +} + +#[test] +fn the_command_exits_2_for_usage_errors() { + let binary = example_binary(); + assert!(binary.exists(), "{} was not built", binary.display()); + for arguments in [&["unknown"][..], &["--fault", "nope", "fulfillment"], &["--workdir"], &[]] { + let output = std::process::Command::new(&binary).args(arguments).output().unwrap(); + assert_eq!(output.status.code(), Some(2), "{arguments:?}: {}", String::from_utf8_lossy(&output.stderr)); + assert!(!output.stderr.is_empty(), "{arguments:?}: no message"); + } +} + +#[test] +fn live_the_command_exits_1_for_a_jacquard_failure() { + let Some(root) = scratch() else { return }; + let output = std::process::Command::new(example_binary()) + .args(["--workdir", root.0.to_str().unwrap(), "--fault", "publish-failed", "fulfillment"]) + .output() + .unwrap(); + assert_eq!(output.status.code(), Some(1), "{}", String::from_utf8_lossy(&output.stderr)); + assert!(String::from_utf8_lossy(&output.stderr).contains("E1613")); + assert!(String::from_utf8_lossy(&output.stdout).contains("\"pendingDeliveries\":1")); +} diff --git a/rust/tests/sdk_live.rs b/rust/tests/sdk_live.rs new file mode 100644 index 0000000..e189550 --- /dev/null +++ b/rust/tests/sdk_live.rs @@ -0,0 +1,449 @@ +//! Live SDK tests against the real Core named by `JACQUARD_BIN` and +//! `JACQUARD_PRELUDE`; each test skips without them. + +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::sync::atomic::{AtomicUsize, Ordering}; + +use jacquard_host::carrier::Cancel; +use jacquard_host::host::{Failure, HandlerContext, HandlerError}; +use jacquard_host::json::{INT_MAX, INT_MIN, Json, decode_object}; +use jacquard_host::sdk::{Dynamic, Error, Function, Operations, PrepareOptions, Program, Runtime, RuntimeOptions}; +use jacquard_host::{record, variants}; + +/// A runtime and a scratch directory, removed on drop. +struct Live { + runtime: Runtime, + root: PathBuf, +} + +impl Drop for Live { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.root); + } +} + +static NEXT: AtomicUsize = AtomicUsize::new(0); + +fn live() -> Option { + if std::env::var_os("JACQUARD_BIN").is_none() || std::env::var_os("JACQUARD_PRELUDE").is_none() { + // CI sets JACQUARD_REQUIRE_LIVE so a misconfigured live job fails. + assert!(std::env::var_os("JACQUARD_REQUIRE_LIVE").is_none(), "live tests need JACQUARD_BIN and JACQUARD_PRELUDE"); + eprintln!("skipped: live tests need JACQUARD_BIN and JACQUARD_PRELUDE"); + return None; + } + let root = std::env::temp_dir().join(format!("jacquard-host-rust-sdk-{}-{}", std::process::id(), NEXT.fetch_add(1, Ordering::SeqCst))); + std::fs::create_dir_all(&root).unwrap(); + Some(Live { runtime: Runtime::new(RuntimeOptions::default()).unwrap(), root }) +} + +fn example(project: &str) -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")).join("../examples").join(project).join("rules.jac") +} + +impl Live { + fn options(&self) -> PrepareOptions { + PrepareOptions { cache_dir: Some(self.root.join("cache")) } + } + + fn prepare(&self, source: &Path) -> Program { + self.runtime.prepare(&[source], &self.options()).unwrap() + } + + fn source(&self, name: &str, text: &str) -> PathBuf { + let path = self.root.join(name); + std::fs::write(&path, text).unwrap(); + path + } +} + +#[test] +fn live_quickstart_calls_invocations_and_saved_frames() { + let Some(s) = live() else { return }; + let shipping = s.prepare(&example("quickstart")).function("shipping-cost").unwrap(); + for (total, expected) in [(35, 5), (75, 0)] { + assert_eq!(shipping.call::((total,)).unwrap(), expected); + } + let invocation = shipping.invoke((35,)).unwrap(); + assert_eq!(invocation.value().unwrap(), Dynamic::Int(5)); + assert_eq!(invocation.render(), "shipping-cost: ok"); + let path = s.root.join("inspection.json"); + invocation.save(&path).unwrap(); + let raw = std::fs::read(&path).unwrap(); + let saved = decode_object(&raw[..raw.len() - 1], 128).unwrap(); + assert_eq!(saved["schema"], Json::from("jacquard-host-inspection-v1")); + assert_eq!(saved["core_frames"].as_array().unwrap().len(), invocation.raw.core_frames.len()); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + assert_eq!(std::fs::metadata(&path).unwrap().permissions().mode() & 0o777, 0o600); + } +} + +struct TriagePolicy { + page: i64, + stale: i64, +} +record!(TriagePolicy = "TriagePolicy" { page, stale }); + +#[derive(Clone)] +struct Incident { + id: String, + service: String, + severity: i64, + age: i64, + visible: bool, + owner: Option, +} +record!(Incident = "Incident" { id, service, severity, age, visible, owner }); + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Urgency { + Page, + Today, + Watch, +} +variants!(Urgency { Page = "PageNow", Today = "WorkToday", Watch = "WatchOnly" }); + +#[derive(Debug, PartialEq, Eq)] +enum Triage { + Routed { id: String, team: String, urgency: Urgency, reasons: Vec }, + InvalidIncident(String), +} +variants!(Triage { + Routed { id, team, urgency, reasons } = "Routed", + InvalidIncident(id) = "InvalidIncident", +}); + +fn incident(i: i64) -> Incident { + Incident { + id: i.to_string(), + service: "api".into(), + severity: i % 5 + 1, + age: i * 3, + visible: i % 2 == 0, + owner: (i % 3 != 0).then(|| "ops".into()), + } +} + +fn triage(s: &Live) -> (Program, Function) { + let program = s.prepare(&example("triage")); + let route = program.function("triage.batch").unwrap(); + (program, route) +} + +#[test] +fn live_records_enums_options_and_explicit_batches() { + let Some(s) = live() else { return }; + let (_, route) = triage(&s); + let policy = TriagePolicy { page: 4, stale: 60 }; + let rows: Vec = (0..64).map(incident).collect(); + let results: Vec = route.map_batches((&policy,), &rows, 8).unwrap(); + assert_eq!(results.len(), rows.len()); + for (row, got) in rows.iter().zip(&results) { + let page = row.visible && row.severity >= 4; + let mut reasons = Vec::new(); + if row.owner.is_none() { + reasons.push("unowned".to_owned()); + } + if row.age >= 60 { + reasons.push("ageing".to_owned()); + } + if page { + reasons.push("customer-impact".to_owned()); + } + let urgency = if page { + Urgency::Page + } else if row.age >= 60 || row.severity == 5 { + Urgency::Today + } else { + Urgency::Watch + }; + let team = row.owner.clone().unwrap_or_else(|| "api-oncall".into()); + assert_eq!(*got, Triage::Routed { id: row.id.clone(), team, urgency, reasons }, "row {}", row.id); + } + let empty: Vec = route.map_batches((&policy,), &[] as &[Incident], 8).unwrap(); + assert!(empty.is_empty()); +} + +#[test] +fn live_invalid_input_is_refused_before_any_worker() { + let Some(s) = live() else { return }; + let mut program = s.prepare(&example("triage")); + // A spawn would now fail as worker_spawn_failed, not as a local refusal. + program.set_worker_command(vec!["/no/such/jacquard".into()]); + let route = program.function("triage.batch").unwrap(); + let policy = TriagePolicy { page: 4, stale: 60 }; + let mut rows: Vec = (0..17).map(incident).collect(); + rows[16].severity = INT_MAX + 1; + let error = route.map_batches::((&policy,), &rows, 8).unwrap_err(); + assert!(error.index.is_none() && error.completed.is_empty()); + assert!( + matches!(&error.error, Error::InvalidArgument(argument) if argument.message.contains("argument[1][0].severity")), + "late invalid row: {error}" + ); + let many = vec![incident(1); 64]; + let error = route.call::>((&policy, &many)).unwrap_err(); + assert!(matches!(error, Error::InvalidArgument(_)), "over-deep single call: {error}"); + let error = route.call::>((&policy, "not a list")).unwrap_err(); + assert!(matches!(&error, Error::InvalidArgument(argument) if argument.message.contains("argument[1]")), "{error}"); + let error = route.call::>((&policy,)).unwrap_err(); + assert!(matches!(&error, Error::InvalidArgument(argument) if argument.message.contains("expected 2 arguments")), "{error}"); +} + +#[test] +fn live_missing_handlers_refuse_binding_and_pure_calls_need_no_services() { + let Some(s) = live() else { return }; + let mut fulfillment = s.prepare(&example("fulfillment")); + let calls = Arc::new(AtomicUsize::new(0)); + let counted = calls.clone(); + let inventory = Operations::new().on("booking-status", move |_: Dynamic| { + counted.fetch_add(1, Ordering::SeqCst); + Ok(Dynamic::unit()) + }); + let error = fulfillment.function_with("fulfillment.place", [("Inventory", inventory.clone())]).unwrap_err(); + let message = error.to_string(); + assert!(matches!(error, Error::Binding { .. }), "{message}"); + assert!(message.contains("inventory.lookup-stock") && message.contains("dispatch.publish-booking"), "{message}"); + assert_eq!(calls.load(Ordering::SeqCst), 0, "a handler ran during binding"); + let typo = Operations::new().on("lookup_stok", |_: String| Ok(())); + let error = fulfillment.function_with("fulfillment.place", [("Inventory", typo)]).unwrap_err(); + assert!(error.to_string().contains("inventory has no operation lookup-stok"), "{error}"); + fulfillment.provide("Inventory", inventory).unwrap(); + let valid = fulfillment.function("order.valid-lines").unwrap(); + assert!(valid.call::((Vec::::new(), Vec::::new())).unwrap()); + assert!(matches!(fulfillment.provide("Unknown", Operations::new()), Err(Error::Binding { .. }))); + assert!(matches!(fulfillment.function("no-such-function"), Err(Error::Binding { .. }))); +} + +#[test] +fn live_unknown_completion_happens_once_with_raw_evidence() { + let Some(s) = live() else { return }; + let fulfillment = s.prepare(&example("fulfillment")); + let calls = Arc::new(AtomicUsize::new(0)); + let counted = calls.clone(); + let warehouse = Operations::new() + .on("booking-status", |_: Dynamic| Ok(Dynamic::variant("BookingPending", []))) + .on("lookup_stock", |_: String| Ok(Dynamic::variant("Stock", [20.into()]))) + .on("reserve-order", move |context: &HandlerContext, _: Dynamic| { + assert!(!context.is_cancelled()); + counted.fetch_add(1, Ordering::SeqCst); + Ok(Dynamic::variant("Booked", ["receipt-1".into()])) + }) + .on("publish-booking", |_: String, _: String| -> Result<(), HandlerError> { + Err(std::io::Error::other("synthetic after commit"))? + }); + let place = fulfillment.function_with("fulfillment.place", [("Inventory", warehouse.clone()), ("Dispatch", warehouse)]).unwrap(); + let data = fulfillment.data(); + let line = data.make("Line", ["sensor".into(), 2.into()]).unwrap(); + let order = data.make("Order", ["one".into(), Dynamic::List(vec![line])]).unwrap(); + let error = place.call::((&order,)).unwrap_err(); + assert_eq!(error.failure(), Some(Failure::OperationFailed), "{error}"); + let result = error.result().unwrap(); + assert_eq!(result.code(), "E1612"); + assert!(result.host_frames.iter().any(|frame| frame.get("kind") == Some(&Json::from("effect_ok")))); + assert_eq!(calls.load(Ordering::SeqCst), 1); + let invocation = place.invoke((&order,)).unwrap(); + let rendered = invocation.render(); + assert!(rendered.starts_with("fulfillment.place: E1612") && rendered.contains(". publish-booking"), "{rendered}"); +} + +struct Job { + id: String, + effort: i64, + value: i64, + latest: i64, +} +record!(Job = "Job" { id, effort, value, latest }); + +#[derive(Debug)] +struct Assignment { + id: String, + slot: i64, +} +record!(Assignment = "Assignment" { id, slot }); + +#[derive(Debug)] +struct Plan { + score: i64, + assignments: Vec, +} +record!(Plan = "Plan" { score, assignments }); + +#[derive(Debug)] +enum Planning { + Planned { plan: Plan }, + InvalidPlan(String), +} +variants!(Planning { Planned { plan } = "Planned", InvalidPlan(reason) = "InvalidPlan" }); + +fn optimum(jobs: &[Job], caps: &[i64]) -> i64 { + let Some((job, rest)) = jobs.split_first() else { return 0 }; + let mut best = optimum(rest, caps); + for slot in 0..(job.latest as usize).min(caps.len()) { + if caps[slot] >= job.effort { + let mut next = caps.to_vec(); + next[slot] -= job.effort; + best = best.max(job.value + optimum(rest, &next)); + } + } + best +} + +#[test] +fn live_planner_matches_an_independent_exhaustive_oracle() { + let Some(s) = live() else { return }; + let solve = s.prepare(&example("planner")).function("planner.plan").unwrap(); + for seed in 0..25i64 { + let jobs: Vec = (0..seed % 6 + 1) + .map(|i| Job { + id: format!("job-{i}"), + effort: (seed + i * 3) % 4 + 1, + value: (seed * 7 + i * 11) % 15, + latest: (seed + i) % 3 + 1, + }) + .collect(); + let caps = [seed % 5 + 1, seed % 3 + 2, 3][..(seed % 3 + 1) as usize].to_vec(); + let Planning::Planned { plan } = solve.call::((&jobs, &caps)).unwrap() else { panic!("seed {seed}: not planned") }; + assert_eq!(plan.score, optimum(&jobs, &caps), "seed {seed}"); + let mut loads = vec![0; caps.len()]; + let mut seen = std::collections::BTreeSet::new(); + let mut score = 0; + for assignment in &plan.assignments { + let job = jobs.iter().find(|job| job.id == assignment.id).expect("a known job"); + assert!( + seen.insert(&assignment.id) && assignment.slot >= 1 && assignment.slot <= job.latest.min(caps.len() as i64), + "{assignment:?}" + ); + loads[assignment.slot as usize - 1] += job.effort; + score += job.value; + } + assert!(loads.iter().zip(&caps).all(|(load, cap)| load <= cap), "seed {seed}: overloaded"); + assert_eq!(score, plan.score, "seed {seed}"); + } +} + +#[test] +fn live_cache_reuse_invalidation_and_corruption() { + let Some(s) = live() else { return }; + let quick = s.prepare(&example("quickstart")); + let commands = s.runtime.core_commands(); + let repeated = s.prepare(&example("quickstart")); + assert_eq!(s.runtime.core_commands(), commands, "a warm preparation ran Core"); + assert_eq!(repeated.directory(), quick.directory()); + let original = std::fs::read_to_string(example("quickstart")).unwrap(); + let changed = s.source("changed.jac", &format!("{}\r\n-- changed\r\n", original.replace('\n', "\r\n"))); + let program = s.prepare(&changed); + let snapshot = std::fs::read_to_string(program.directory().join("source.jac")).unwrap(); + assert!(program.directory() != quick.directory() && snapshot.contains("\r\n"), "not a byte-exact new entry"); + std::fs::write(program.directory().join("store").join("unexpected"), "corrupt").unwrap(); + let error = s.runtime.prepare(&[&changed], &s.options()).expect_err("a corrupt cache was accepted"); + assert!(matches!(error, Error::Preparation(_)), "{error}"); + assert!(matches!(s.runtime.prepare(&[s.source("rules.txt", "")], &s.options()), Err(Error::Preparation(_)))); +} + +#[test] +fn live_full_int_range_reals_and_nested_options() { + let Some(s) = live() else { return }; + let program = s.prepare(&s.source( + "values.jac", + "value.int : (Int) ->{} Int\nvalue.int(x) = x\nvalue.real : (Real) ->{} Real\nvalue.real(x) = x\nvalue.option : (Option (Option Int)) ->{} Option (Option Int)\nvalue.option(x) = x\n", + )); + let integer = program.function("value.int").unwrap(); + for value in [INT_MIN, INT_MAX] { + assert_eq!(integer.call::((value,)).unwrap(), value); + } + assert!(matches!(integer.call::((INT_MAX + 1,)), Err(Error::InvalidArgument(_)))); + assert!(matches!(integer.call::((1.5,)), Err(Error::InvalidArgument(_)))); + assert!(matches!(integer.call::((300,)), Err(Error::Binding { result: Some(_), .. }))); + let real = program.function("value.real").unwrap(); + for value in [1.0, -0.0, f64::INFINITY, f64::NAN] { + assert_eq!(real.call::((value,)).unwrap().to_bits(), value.to_bits()); + } + let option = program.function("value.option").unwrap(); + assert_eq!(option.call::>>((None::>,)).unwrap(), None); + assert_eq!(option.call::>>((Some(None::),)).unwrap(), Some(None)); + assert_eq!(option.call::>>((Some(Some(3)),)).unwrap(), Some(Some(3))); + let nested = Dynamic::Option(Some(Box::new(Dynamic::Option(None)))); + assert_eq!(option.call::((&nested,)).unwrap(), nested); +} + +#[test] +fn live_spawn_failures_and_cancellation() { + let Some(s) = live() else { return }; + let mut quick = s.prepare(&example("quickstart")); + let shipping = quick.function("shipping-cost").unwrap(); + quick.set_worker_command(vec!["/no/such/jacquard".into()]); + let error = quick.function("shipping-cost").unwrap().call::((10,)).unwrap_err(); + assert_eq!(error.failure(), Some(Failure::WorkerLost), "{error}"); + assert_eq!(error.result().unwrap().host_evidence.get("reason"), Some(&Json::from("worker_spawn_failed"))); + let cancel = Cancel::new(); + cancel.cancel(); + let invocation = shipping.invoke_with((10,), &cancel).unwrap(); + assert_eq!(invocation.raw.host_evidence.get("reason"), Some(&Json::from("aborted"))); + assert!(invocation.error().is_some()); +} + +#[test] +fn live_output_overflow_fails_once_without_adaptive_batching() { + let Some(s) = live() else { return }; + let (_, route) = triage(&s); + let rows: Vec = + (0..29).map(|i| Incident { id: i.to_string(), service: "api".into(), severity: 5, age: 90, visible: true, owner: None }).collect(); + let error = route.map_batches::((TriagePolicy { page: 4, stale: 60 },), &rows, 29).unwrap_err(); + assert_eq!(error.index, Some(0), "{error}"); + assert!(error.completed.is_empty()); + assert_eq!(error.error.result().map(|result| result.code()), Some("E1602".into())); +} + +#[test] +fn live_result_mapping_failure_retains_the_successful_result() { + let Some(s) = live() else { return }; + let shipping = s.prepare(&example("quickstart")).function("shipping-cost").unwrap(); + let invocation = shipping.invoke((35,)).unwrap(); + assert!(invocation.ok()); + let error = invocation.decode::().unwrap_err(); + assert!(matches!(&error, Error::Binding { result: Some(result), .. } if result.ok()), "{error}"); +} + +fn copy_tree(from: &Path, to: &Path) { + std::fs::create_dir_all(to).unwrap(); + for entry in std::fs::read_dir(from).unwrap() { + let entry = entry.unwrap(); + let target = to.join(entry.file_name()); + if entry.file_type().unwrap().is_dir() { + copy_tree(&entry.path(), &target); + } else { + std::fs::copy(entry.path(), target).unwrap(); + } + } +} + +#[test] +fn live_prelude_contents_key_the_cache_and_concurrent_preparation_publishes_once() { + let Some(s) = live() else { return }; + let quick = s.prepare(&example("quickstart")); + let copy = s.root.join("prelude"); + copy_tree(Path::new(&std::env::var_os("JACQUARD_PRELUDE").unwrap()), ©); + let copied = Runtime::new(RuntimeOptions { prelude: Some(copy.clone()), ..RuntimeOptions::default() }).unwrap(); + let before = copied.prepare(&[example("quickstart")], &s.options()).unwrap(); + assert_eq!(before.directory(), quick.directory(), "an identical prelude copy should share the entry"); + let license = copy.join("LICENSE.md"); + let mut text = std::fs::read(&license).unwrap(); + text.push(b'\n'); + std::fs::write(&license, text).unwrap(); + let after = copied.prepare(&[example("quickstart")], &s.options()).unwrap(); + assert_ne!(after.directory(), before.directory(), "a prelude change did not change the cache key"); + let source = s + .source("concurrent.jac", "shipping-cost : (Int) ->{} Int\nshipping-cost(total) = if lt(total, 50) then 5 else 0\n-- concurrent\n"); + let programs: Vec = std::thread::scope(|scope| { + let workers: Vec<_> = (0..2).map(|_| scope.spawn(|| s.runtime.prepare(&[&source], &s.options()).unwrap())).collect(); + workers.into_iter().map(|worker| worker.join().unwrap()).collect() + }); + assert_eq!(programs[0].directory(), programs[1].directory()); + assert_eq!(programs[1].function("shipping-cost").unwrap().call::((35,)).unwrap(), 5); + for entry in std::fs::read_dir(s.root.join("cache")).unwrap() { + let name = entry.unwrap().file_name(); + assert!(!name.to_string_lossy().starts_with("prepare-"), "staging directory left behind: {name:?}"); + } +}