diff --git a/.agents/skills/layerfs-js-sdk/SKILL.md b/.agents/skills/layerfs-js-sdk/SKILL.md index 2d67c5be08..071d681e87 100644 --- a/.agents/skills/layerfs-js-sdk/SKILL.md +++ b/.agents/skills/layerfs-js-sdk/SKILL.md @@ -285,7 +285,7 @@ For Rust projects, follow the [Cargo workflow](references/rules/cargo.md). ## Calls -85 calls in twelve families. Each family's page has arguments, results, refusals +84 calls in twelve families. Each family's page has arguments, results, refusals and examples. | Family | Calls | diff --git a/.agents/skills/layerfs-js-sdk/references/methods/store.md b/.agents/skills/layerfs-js-sdk/references/methods/store.md index 19d1bc2530..9cfeec57ae 100644 --- a/.agents/skills/layerfs-js-sdk/references/methods/store.md +++ b/.agents/skills/layerfs-js-sdk/references/methods/store.md @@ -10,12 +10,11 @@ ```js store.list() store.get(name) -store.status(name) +store.status(name, { machine }) store.stats(name) store.create(name) store.connect(name, url, { password, synchronous_commit }) -store.attach(name, machine, { url, password }) -store.detach(name, machine) +store.update(name, { machine, url, password, synchronous_commit, clear }) store.push(name, machine, { on_progress }) store.pull(name, { on_progress }) store.remove(name) @@ -25,7 +24,7 @@ store.disconnect(name) A Store holds projects and all their history. The verb says the kind: `create` makes a sqlite Store (one file, on one machine at a time; it starts on -`native`); `connect` attaches a PostgreSQL database. Every sqlite Store is +`native`); `connect` connects a PostgreSQL database. Every sqlite Store is Disposable (WAL, `synchronous=OFF`), the only allowed profile. ## Reads @@ -80,9 +79,8 @@ Store on another machine is read through its worker; a stopped worker is | Call | What it does | Notes | | --- | --- | --- | | `create(name)` | An empty sqlite Store on `native` | Live result: the record above, with `projects: []`. `Exists` for a taken name | -| `connect(name, url, …)` | Attaches a PostgreSQL database | `url` is `postgresql://@:/[?sslmode=…]` without a password (one in the user part or in any query parameter whose name contains `password` is `InvalidInput`); `password` is a reference, `env:LAYERFS_` or `file:/path`. `synchronous_commit` defaults to `local`. The same database under a second name is `Exists` | -| `attach(name, machine, …)` | Lets a machine run Sandboxes of a PostgreSQL Store | Checks from that machine that it reaches the database and its role may write; `url` and `password` replace the Store's for that machine, either alone. `InvalidInput` for sqlite; `Unsupported` when the platform runs no PostgreSQL Sandboxes; `Unavailable` (not reached, machine stopped); `Denied` (login or privilege); `Conflict` while the Store is disconnected or reconnected | -| `detach(name, machine)` | Stops a machine running Sandboxes of a PostgreSQL Store | Dials nothing; detaching `native` is allowed. `InvalidInput` (sqlite, or not attached there); `InUse` while a Sandbox of the Store is recorded on the machine | +| `connect(name, url, …)` | Connects a PostgreSQL database | `url` is `postgresql://@:/[?sslmode=…]` without a password (one in the user part or in any query parameter whose name contains `password` is `InvalidInput`); `password` is a reference, `env:LAYERFS_` or `file:/path`. `synchronous_commit` defaults to `local`. The same database under a second name is `Exists` | +| `update(name, …)` | Changes a PostgreSQL Store's connection, or one machine's own | Without `machine`: sets the Store's `url`, `password` reference and `synchronous_commit`; a new URL or password must reach the same Store. With `machine`: `url` and/or `password` become that machine's own connection, checked from the machine first; `clear: true` removes it. A PostgreSQL Store's Sandboxes run on every supported machine with no step before `sandbox.create`. `InvalidInput` for sqlite or nothing to update; `Unsupported` when the platform runs no PostgreSQL Sandboxes; `Unavailable` (not reached, machine stopped, another Store's database); `Denied` (login or privilege); `Conflict` while the Store is disconnected or reconnected | | `push(name, machine, …)` | Sends a sqlite Store's file from `native` to `machine` | `InvalidInput` unless the Store is on `native` (pull it first). Progress `{ phase, done, total }` on `on_progress` / `--progress`: `done` and `total` are bytes during `send`, null in the other phases (`check`, `snapshot`, `verify`, `install`, `record`, `retire`). `InUse` while the Store is held (Sandbox records, processes with it open; `detail.related` lists them as `{ kind, name }`); `Capacity` when the target has less free disk than the Store plus a headroom of 1 GiB (or a tenth of the Store), checked before anything changes; `Unavailable` naming `layerfs machine start ` for a stopped machine; while it runs, Store calls and operations on it are `Busy`, from the caller's process too. An uncertain push whose process is gone (it ended during install or record, or a step's answer was lost) leaves every later call `Unresolved`, naming `store status` and the push that clears it: a new `push` (or `pull`) first deletes the copy on the machine the record does not name, then runs. A push whose `retire` did not finish leaves the Store in use on `machine` and `last_transfer.retired: false`; the next `pull` deletes the old copy first | | `pull(name, …)` | Brings a sqlite Store's file from its machine to `native` | `InvalidInput` when it is already on `native`; otherwise as `push`. Between two machines: `pull`, then `push` | | `remove(name)` | Deletes a sqlite Store with every project in it | Returns `{ removed }`. `NotFound` for an unknown name (live); `Unavailable` for a Store whose machine is unknown; `Busy` while a live process pushes, pulls or removes it. It clears an `Unresolved` Store. Backups stay | diff --git a/api/README.md b/api/README.md index db6aff2bbe..d3f038d84c 100644 --- a/api/README.md +++ b/api/README.md @@ -10,7 +10,7 @@ CLI package `@layerfs/cli` (in [`cli/`](cli)), the Python package (in the private stdio MCP server [`mcp/`](mcp) (`@layerfs/mcp`: one generated tool per call, `work` and `admin` profiles). The design is in [docs/architecture/api](../docs/architecture/api/README.md). The package is the -library's call table as twelve families of functions, 85 calls in all: +library's call table as twelve families of functions, 84 calls in all: | Family | Calls | | --- | --- | @@ -22,7 +22,7 @@ library's call table as twelve families of functions, 85 calls in all: | [`sandbox`](../docs/architecture/api/sdk/sandbox.md) | `create`, `list`, `get`, `close`, `files` | | [`workspace`](../docs/architecture/api/sdk/workspace.md) | `mount`, `exec`, `exec_cancel`, `commit`, `unmount`, `status` | | [`machine`](../docs/architecture/api/sdk/machine.md) | `list`, `platforms`, `get`, `images`, `load`, `prune`, `status`, `stats`, `space`, `files`, `start`, `stop`, `create`, `remove`, `update` | -| [`store`](../docs/architecture/api/sdk/store.md) | `list`, `get`, `status`, `stats`, `create`, `connect`, `attach`, `detach`, `push`, `pull`, `remove`, `compact`, `disconnect` | +| [`store`](../docs/architecture/api/sdk/store.md) | `list`, `get`, `status`, `stats`, `create`, `connect`, `update`, `push`, `pull`, `remove`, `compact`, `disconnect` | | [`backup`](../docs/architecture/api/sdk/backup.md) | `connect`, `disconnect`, `list`, `create`, `restore` | | [`logs`](../docs/architecture/api/sdk/logs.md) | `list`, `reset`, `machine`, `sandbox` | | [`doctor`](../docs/architecture/api/sdk/doctor.md) | `fix` | diff --git a/api/crates/layerfs-api/src/lib.rs b/api/crates/layerfs-api/src/lib.rs index 26a8d5518d..04b89f2174 100644 --- a/api/crates/layerfs-api/src/lib.rs +++ b/api/crates/layerfs-api/src/lib.rs @@ -79,7 +79,7 @@ pub use layerfs_sandbox::{ BoundWorkspace, MountFailure, MountedWorkspace, Unmounted, WorkspaceApi, }; -/// The twelve families, in the order help lists them: 85 calls, the same +/// The twelve families, in the order help lists them: 84 calls, the same /// from the `layerfs` binary, Node and Python. pub fn families() -> Families { &[ diff --git a/api/crates/layerfs-api/src/store/attach.rs b/api/crates/layerfs-api/src/store/attach.rs deleted file mode 100644 index 71607db205..0000000000 --- a/api/crates/layerfs-api/src/store/attach.rs +++ /dev/null @@ -1,32 +0,0 @@ -//! `store.attach`: let a machine run Sandboxes of a PostgreSQL Store, after -//! checking from that machine that it reaches the Store's database. -use crate::calls::table::Kind::{Name, Text}; -use crate::calls::table::{a, call, opt, Call}; -use crate::calls::Context; -use layerfs_bridge::control::SdkValue; -use layerfs_control::store::StoreAttach; -use layerfs_home::names::{MachineName, StoreName}; -use layerfs_home::outcome::Outcome; - -/// Its table entry. -pub(crate) const CALL: Call = call( - "store", - "attach", - &[a("name", Name), a("machine", Name)], - &[opt("url", Text), opt("password", Text)], - "let a machine run Sandboxes of a PostgreSQL Store, after checking it reaches the server", - run, -); - -/// Runs `store.attach`. -fn run(context: &mut Context<'_>, args: &SdkValue) -> Outcome { - let text = |name: &str| args.get_text(name).map(str::to_owned); - let request = StoreAttach { - name: StoreName::parse(args.get_text("name").unwrap_or(""))?, - machine: MachineName::parse(args.get_text("machine").unwrap_or(""))?, - url: text("url"), - password: text("password"), - }; - let (home, events) = context.parts(); - layerfs_control::store::attach(home, events, &request) -} diff --git a/api/crates/layerfs-api/src/store/connect.rs b/api/crates/layerfs-api/src/store/connect.rs index c8aee1b1e4..2a4e48ce93 100644 --- a/api/crates/layerfs-api/src/store/connect.rs +++ b/api/crates/layerfs-api/src/store/connect.rs @@ -1,4 +1,4 @@ -//! `store.connect`: attach a PostgreSQL database as a Store. +//! `store.connect`: connect a PostgreSQL database as a Store. use crate::calls::table::Kind::{Name, Text}; use crate::calls::table::{a, call, opt, Call}; use crate::calls::Context; @@ -13,7 +13,7 @@ pub(crate) const CALL: Call = call( "connect", &[a("name", Name), a("url", Text)], &[opt("password", Text), opt("synchronous_commit", Text)], - "attach a PostgreSQL database as a Store", + "connect a PostgreSQL database as a Store", run, ); diff --git a/api/crates/layerfs-api/src/store/detach.rs b/api/crates/layerfs-api/src/store/detach.rs deleted file mode 100644 index 5fa84de5b8..0000000000 --- a/api/crates/layerfs-api/src/store/detach.rs +++ /dev/null @@ -1,24 +0,0 @@ -//! `store.detach`: stop a machine running Sandboxes of a PostgreSQL Store. -use crate::calls::table::Kind::Name; -use crate::calls::table::{a, call, Call}; -use crate::calls::Context; -use layerfs_bridge::control::SdkValue; -use layerfs_home::names::{MachineName, StoreName}; -use layerfs_home::outcome::Outcome; - -/// Its table entry. -pub(crate) const CALL: Call = call( - "store", - "detach", - &[a("name", Name), a("machine", Name)], - &[], - "stop a machine running Sandboxes of a PostgreSQL Store", - run, -); - -/// Runs `store.detach`. -fn run(context: &mut Context<'_>, args: &SdkValue) -> Outcome { - let name = StoreName::parse(args.get_text("name").unwrap_or(""))?; - let machine = MachineName::parse(args.get_text("machine").unwrap_or(""))?; - layerfs_control::store::detach(&context.home, &name, &machine) -} diff --git a/api/crates/layerfs-api/src/store/mod.rs b/api/crates/layerfs-api/src/store/mod.rs index 2142e041ea..0722c4037f 100644 --- a/api/crates/layerfs-api/src/store/mod.rs +++ b/api/crates/layerfs-api/src/store/mod.rs @@ -1,10 +1,9 @@ -//! The `store` family: 13 calls, one file each (H1: push and pull replace -//! move; attach and detach supersede #536 C4). -mod attach; +//! The `store` family: 12 calls, one file each (H1: push and pull replace +//! move; a PostgreSQL Store runs on every machine, #536 C4, and `update` +//! gives one its own connection, H9). mod compact; mod connect; mod create; -mod detach; mod disconnect; mod get; mod list; @@ -13,6 +12,7 @@ mod push; mod remove; mod stats; mod status; +mod update; use crate::calls::Calls; @@ -25,8 +25,7 @@ pub(crate) fn calls() -> Calls { stats::CALL, create::CALL, connect::CALL, - attach::CALL, - detach::CALL, + update::CALL, push::CALL, pull::CALL, remove::CALL, diff --git a/api/crates/layerfs-api/src/store/status.rs b/api/crates/layerfs-api/src/store/status.rs index 6913105880..684665fa5f 100644 --- a/api/crates/layerfs-api/src/store/status.rs +++ b/api/crates/layerfs-api/src/store/status.rs @@ -1,9 +1,10 @@ -//! `store.status`: a Store's live state. +//! `store.status`: a Store's live state, and on request the check of one +//! machine's connection to it. use crate::calls::table::Kind::Name; -use crate::calls::table::{a, call, Call}; +use crate::calls::table::{a, call, opt, Call}; use crate::calls::Context; use layerfs_bridge::control::SdkValue; -use layerfs_home::names::StoreName; +use layerfs_home::names::{MachineName, StoreName}; use layerfs_home::outcome::Outcome; /// Its table entry. @@ -11,13 +12,18 @@ pub(crate) const CALL: Call = call( "store", "status", &[a("name", Name)], - &[], - "a Store's live state", + &[opt("machine", Name)], + "a Store's live state; with a machine, whether it reaches a PostgreSQL Store", run, ); /// Runs `store.status`. fn run(context: &mut Context<'_>, args: &SdkValue) -> Outcome { let name = StoreName::parse(args.get_text("name").unwrap_or(""))?; - layerfs_store::sqlite::family::status(&context.home, &name) + let machine = args + .get_text("machine") + .map(MachineName::parse) + .transpose()?; + let (home, events) = context.parts(); + layerfs_control::store::status(home, events, &name, machine.as_ref()) } diff --git a/api/crates/layerfs-api/src/store/update.rs b/api/crates/layerfs-api/src/store/update.rs new file mode 100644 index 0000000000..7c04d72756 --- /dev/null +++ b/api/crates/layerfs-api/src/store/update.rs @@ -0,0 +1,43 @@ +//! `store.update`: change a PostgreSQL Store's connection, or the one a +//! machine alone dials. +use crate::calls::table::Kind::{Bool, Name, Text}; +use crate::calls::table::{a, call, opt, Call}; +use crate::calls::Context; +use layerfs_bridge::control::SdkValue; +use layerfs_control::store::StoreUpdate; +use layerfs_home::names::{MachineName, StoreName}; +use layerfs_home::outcome::Outcome; + +/// Its table entry. +pub(crate) const CALL: Call = call( + "store", + "update", + &[a("name", Name)], + &[ + opt("machine", Name), + opt("url", Text), + opt("password", Text), + opt("synchronous_commit", Text), + opt("clear", Bool), + ], + "change a PostgreSQL Store's connection, or the one a machine alone dials", + run, +); + +/// Runs `store.update`. +fn run(context: &mut Context<'_>, args: &SdkValue) -> Outcome { + let text = |name: &str| args.get_text(name).map(str::to_owned); + let request = StoreUpdate { + name: StoreName::parse(args.get_text("name").unwrap_or(""))?, + machine: args + .get_text("machine") + .map(MachineName::parse) + .transpose()?, + url: text("url"), + password: text("password"), + synchronous_commit: text("synchronous_commit"), + clear: args.get_bool("clear"), + }; + let (home, events) = context.parts(); + layerfs_control::store::update(home, events, &request) +} diff --git a/api/crates/layerfs-api/tests/calls_table.rs b/api/crates/layerfs-api/tests/calls_table.rs index a94f20ac97..2859fa9abe 100644 --- a/api/crates/layerfs-api/tests/calls_table.rs +++ b/api/crates/layerfs-api/tests/calls_table.rs @@ -1,4 +1,4 @@ -//! The call table and its dispatcher (#533 step 2): 85 calls in twelve +//! The call table and its dispatcher (#533 step 2): 84 calls in twelve //! families, arguments checked before any effect, one test per family that //! a valid call reaches its family function and a wrong one is refused. use layerfs_api::{families, json, keys, size, Call, Kind}; @@ -51,9 +51,9 @@ fn code(name: &str, args: SdkValue) -> String { } #[test] -fn the_table_has_twelve_families_and_85_calls() { +fn the_table_has_twelve_families_and_84_calls() { let all: Vec<&Call> = layerfs_api::calls(); - assert_eq!(all.len(), 85); + assert_eq!(all.len(), 84); // each family holds only its own calls, and has at least one for family in families() { assert!(!family.calls().is_empty(), "{}", family.name); @@ -66,7 +66,7 @@ fn the_table_has_twelve_families_and_85_calls() { let mut names: Vec<_> = all.iter().map(|c| c.name()).collect(); names.sort(); names.dedup(); - assert_eq!(names.len(), 85, "call names are unique"); + assert_eq!(names.len(), 84, "call names are unique"); // an optional positional is only ever followed by optional ones (D7) for c in &all { let first_optional = c.args.iter().position(|a| a.optional); @@ -75,7 +75,7 @@ fn the_table_has_twelve_families_and_85_calls() { } } let counts: Vec = families().iter().map(|f| f.calls().len()).collect(); - assert_eq!(counts, [2, 7, 9, 3, 15, 5, 6, 15, 13, 5, 4, 1]); + assert_eq!(counts, [2, 7, 9, 3, 15, 5, 6, 15, 12, 5, 4, 1]); } #[test] diff --git a/api/crates/layerfs-api/tests/postgres_attach.rs b/api/crates/layerfs-api/tests/postgres_attach.rs deleted file mode 100644 index 4e7df658fd..0000000000 --- a/api/crates/layerfs-api/tests/postgres_attach.rs +++ /dev/null @@ -1,274 +0,0 @@ -//! `store.attach` and `store.detach` (supersede #536 C4, take up H9): a -//! PostgreSQL Store's Sandboxes run on the machines it is attached to, -//! each with the Store's connection or one of its own, checked from that -//! machine before it is recorded. -//! -//! The first test needs no server: its URLs name 127.0.0.1 port 1, where -//! nothing listens. The second needs LAYERFS_TEST_PG_URL (a server URL -//! without a password, to a database with no `layerfs` schema; the test -//! leaves one behind) and LAYERFS_TEST_PG_PASSWORD; with -//! LAYERFS_TEST_PG_LIMITED_URL and LAYERFS_TEST_PG_LIMITED_PASSWORD (a role -//! on that database that may read LayerFS's tables but not write them) it -//! also checks the privilege refusal. Without them it says so and passes. -#![cfg(feature = "machine")] -use layerfs_api::home::{self, paths, write_record}; -use layerfs_bridge::control::SdkValue; -use std::os::unix::fs::PermissionsExt; -use std::path::{Path, PathBuf}; - -fn tempdir(tag: &str) -> PathBuf { - let dir = std::env::temp_dir().join(format!("layerfs-pgattach-{tag}-{}", std::process::id())); - let _ = std::fs::remove_dir_all(&dir); - std::fs::create_dir_all(&dir).unwrap(); - std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)).unwrap(); - dir -} - -fn t(s: &str) -> SdkValue { - SdkValue::text(s) -} - -fn call(home: &Path, name: &str, args: &[(&str, SdkValue)]) -> Result { - let args = SdkValue::Map( - args.iter() - .map(|(n, v)| ((*n).to_owned(), v.clone())) - .collect(), - ); - layerfs_api::call(Some(home), name, args, &mut |_| {}).map_err(|r| (r.code, r.message)) -} - -fn ok(home: &Path, name: &str, args: &[(&str, SdkValue)]) -> SdkValue { - call(home, name, args).unwrap_or_else(|e| panic!("{name}: {e:?}")) -} - -fn machines(store: &SdkValue) -> Vec { - match store.get("machines") { - Some(SdkValue::Map(machines)) => machines.iter().map(|(m, _)| m.clone()).collect(), - other => panic!("no machines: {other:?}"), - } -} - -fn record_text(home: &Path, store: &str) -> String { - std::fs::read_to_string(paths::store_record(home, store)).unwrap() -} - -#[test] -fn attach_and_detach_refusals_need_no_server() { - let home = tempdir("refusals"); - home::open(&home).unwrap(); - ok(&home, "machine.list", &[]); - // a record written before attachments: attached to native alone - write_record( - &paths::store_record(&home, "pg"), - &SdkValue::map([ - ("name", t("pg")), - ("kind", t("postgresql")), - ("url", t("postgresql://alice@127.0.0.1:1/app")), - ("schema", t("layerfs")), - ("synchronous_commit", t("off")), - ("binding", SdkValue::Bytes(vec![7; 32])), - ("cursor", SdkValue::Bytes(vec![9; 32])), - ]), - ) - .unwrap(); - write_record( - &paths::project_record(&home, "pg", "app"), - &SdkValue::map([("name", t("app"))]), - ) - .unwrap(); - write_record( - &paths::store_record(&home, "shop"), - &SdkValue::map([("name", t("shop")), ("kind", t("sqlite"))]), - ) - .unwrap(); - let shown = ok(&home, "store.get", &[("name", t("pg"))]); - assert_eq!(machines(&shown), ["native"]); - let native = ok(&home, "machine.get", &[("name", t("native"))]); - assert_eq!( - native.get("postgresql"), - Some(&SdkValue::List(vec![t("pg")])) - ); - let before = record_text(&home, "pg"); - - let attach = |args: &[(&str, SdkValue)]| call(&home, "store.attach", args).unwrap_err(); - let (code, message) = attach(&[("name", t("nope")), ("machine", t("native"))]); - assert_eq!(code, "NotFound", "{message}"); - let (code, message) = attach(&[("name", t("shop")), ("machine", t("native"))]); - assert_eq!(code, "InvalidInput", "{message}"); - assert!(message.contains("`store push shop `"), "{message}"); - let (code, message) = attach(&[("name", t("pg")), ("machine", t("nowhere"))]); - assert_eq!(code, "NotFound", "{message}"); - let (code, message) = attach(&[ - ("name", t("pg")), - ("machine", t("native")), - ("url", t("postgresql://alice:secret@127.0.0.1:1/app")), - ]); - assert_eq!(code, "InvalidInput", "{message}"); - assert!(!message.contains("secret"), "{message}"); - let (code, message) = attach(&[ - ("name", t("pg")), - ("machine", t("native")), - ("password", t("env:LAYERFS_TEST_ATTACH_UNSET_PASSWORD")), - ]); - // the reference is resolved before any connection, as connect's is - assert!( - message.contains("LAYERFS_TEST_ATTACH_UNSET_PASSWORD is not set"), - "{code}: {message}" - ); - // the check from native finds no server: nothing is recorded - let (code, message) = attach(&[("name", t("pg")), ("machine", t("native"))]); - if true { - assert_eq!(code, "Unavailable", "{message}"); - assert!( - message.starts_with( - "machine native cannot reach PostgreSQL Store pg's server at 127.0.0.1: " - ), - "{message}" - ); - } else { - assert_eq!(code, "Unsupported", "{message}"); - } - assert_eq!( - record_text(&home, "pg"), - before, - "a refused attach writes nothing" - ); - - let detach = |machine: &str| { - call( - &home, - "store.detach", - &[("name", t("pg")), ("machine", t(machine))], - ) - }; - let (code, message) = detach("docker").unwrap_err(); - assert_eq!(code, "InvalidInput", "{message}"); - assert_eq!(message, "Store pg is not attached to machine docker"); - // a Sandbox recorded on native holds the attachment - write_record( - &paths::sandbox_record(&home, "s"), - &SdkValue::map([ - ("v", SdkValue::Int(1)), - ("name", t("s")), - ("project", t("pg/app")), - ("machine", t("native")), - ]), - ) - .unwrap(); - let (code, message) = detach("native").unwrap_err(); - assert_eq!(code, "InUse", "{message}"); - std::fs::remove_dir_all(paths::sandbox_dir(&home, "s")).unwrap(); - let shown = detach("native").unwrap(); - assert!(machines(&shown).is_empty(), "{shown:?}"); - let native = ok(&home, "machine.get", &[("name", t("native"))]); - assert_eq!(native.get("postgresql"), Some(&SdkValue::List(Vec::new()))); - // a machine it is not attached to runs none of its Sandboxes - let (code, message) = call( - &home, - "sandbox.create", - &[("project", t("pg/app")), ("name", t("t"))], - ) - .unwrap_err(); - assert_eq!(code, "Unsupported", "{message}"); - assert_eq!( - message, - "PostgreSQL Store pg is not attached to machine native; `layerfs store attach pg native`" - ); - assert!(!paths::sandbox_dir(&home, "t").exists()); - let _ = std::fs::remove_dir_all(home); -} - -#[test] -fn attach_checks_the_machine_s_connection_before_it_records_it() { - let (Ok(url), Ok(_)) = ( - std::env::var("LAYERFS_TEST_PG_URL"), - std::env::var("LAYERFS_TEST_PG_PASSWORD"), - ) else { - eprintln!("NOT_RUN: LAYERFS_TEST_PG_URL / LAYERFS_TEST_PG_PASSWORD unset"); - return; - }; - let home = tempdir("live"); - home::open(&home).unwrap(); - let connected = ok( - &home, - "store.connect", - &[ - ("name", t("pg")), - ("url", t(&url)), - ("password", t("env:LAYERFS_TEST_PG_PASSWORD")), - ], - ); - // connect attaches native, its check recorded - let native = connected.get("machines").and_then(|m| m.get("native")); - let checked = native.and_then(|n| n.get("checked")); - assert!( - checked.and_then(|c| c.get_text("server_version")).is_some(), - "{connected:?}" - ); - // native again with a connection of its own: the same server, the - // password by reference; the record holds no password - let shown = ok( - &home, - "store.attach", - &[ - ("name", t("pg")), - ("machine", t("native")), - ("url", t(&url)), - ("password", t("env:LAYERFS_TEST_PG_PASSWORD")), - ], - ); - let native = shown.get("machines").and_then(|m| m.get("native")).unwrap(); - assert_eq!(native.get_text("url"), Some(url.as_str())); - assert_eq!( - native.get_text("password"), - Some("env:LAYERFS_TEST_PG_PASSWORD") - ); - let secret = std::env::var("LAYERFS_TEST_PG_PASSWORD").unwrap(); - assert!(!record_text(&home, "pg").contains(&secret)); - // another database is not this Store: refused, nothing recorded - let before = record_text(&home, "pg"); - let other = match url.rsplit_once('/') { - Some((server, _)) => format!("{server}/postgres"), - None => url.clone(), - }; - let (code, message) = call( - &home, - "store.attach", - &[ - ("name", t("pg")), - ("machine", t("native")), - ("url", t(&other)), - ("password", t("env:LAYERFS_TEST_PG_PASSWORD")), - ], - ) - .unwrap_err(); - assert_eq!(code, "Unavailable", "{message}"); - assert!(message.contains("holds another Store than pg"), "{message}"); - // the server answered: no "cannot reach" and no reach hint - assert!(!message.contains("cannot reach"), "{message}"); - assert_eq!(record_text(&home, "pg"), before); - // a role that may not write is refused naming what it lacks - if let (Ok(limited), Ok(_)) = ( - std::env::var("LAYERFS_TEST_PG_LIMITED_URL"), - std::env::var("LAYERFS_TEST_PG_LIMITED_PASSWORD"), - ) { - let (code, message) = call( - &home, - "store.attach", - &[ - ("name", t("pg")), - ("machine", t("native")), - ("url", t(&limited)), - ("password", t("env:LAYERFS_TEST_PG_LIMITED_PASSWORD")), - ], - ) - .unwrap_err(); - assert_eq!(code, "Denied", "{message}"); - assert!(message.contains(" lacks "), "{message}"); - assert_eq!(record_text(&home, "pg"), before); - } else { - eprintln!("NOT_RUN (privileges): LAYERFS_TEST_PG_LIMITED_URL unset"); - } - ok(&home, "store.disconnect", &[("name", t("pg"))]); - let _ = std::fs::remove_dir_all(home); -} diff --git a/api/crates/layerfs-api/tests/postgres_update.rs b/api/crates/layerfs-api/tests/postgres_update.rs new file mode 100644 index 0000000000..c51eb20975 --- /dev/null +++ b/api/crates/layerfs-api/tests/postgres_update.rs @@ -0,0 +1,357 @@ +//! A PostgreSQL Store's connections (#536 C4 again, with H9): its +//! Sandboxes run on every machine whose platform runs them, with no step +//! before the create; `store.update` changes the Store's own connection, or +//! with `--machine` gives one machine its own, checked from that machine +//! before it is saved; `store.status --machine` runs that check and +//! records nothing. +//! +//! The first test needs no server: its URLs name 127.0.0.1 port 1, where +//! nothing listens. The second needs LAYERFS_TEST_PG_URL (a server URL +//! without a password, to a database with no `layerfs` schema; the test +//! leaves one behind) and LAYERFS_TEST_PG_PASSWORD; with +//! LAYERFS_TEST_PG_LIMITED_URL and LAYERFS_TEST_PG_LIMITED_PASSWORD (a role +//! on that database that may read LayerFS's tables but not write them) it +//! also checks the privilege refusal. Without them it says so and passes. +#![cfg(feature = "machine")] +use layerfs_api::home::{self, paths, write_record}; +use layerfs_bridge::control::SdkValue; +use std::os::unix::fs::PermissionsExt; +use std::path::{Path, PathBuf}; + +fn tempdir(tag: &str) -> PathBuf { + let dir = std::env::temp_dir().join(format!("layerfs-pgupdate-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)).unwrap(); + dir +} + +fn t(s: &str) -> SdkValue { + SdkValue::text(s) +} + +fn call(home: &Path, name: &str, args: &[(&str, SdkValue)]) -> Result { + let args = SdkValue::Map( + args.iter() + .map(|(n, v)| ((*n).to_owned(), v.clone())) + .collect(), + ); + layerfs_api::call(Some(home), name, args, &mut |_| {}).map_err(|r| (r.code, r.message)) +} + +fn ok(home: &Path, name: &str, args: &[(&str, SdkValue)]) -> SdkValue { + call(home, name, args).unwrap_or_else(|e| panic!("{name}: {e:?}")) +} + +fn own(store: &SdkValue) -> Vec { + match store.get("machines") { + Some(SdkValue::Map(machines)) => machines.iter().map(|(m, _)| m.clone()).collect(), + None => Vec::new(), + other => panic!("machines: {other:?}"), + } +} + +fn record_text(home: &Path, store: &str) -> String { + std::fs::read_to_string(paths::store_record(home, store)).unwrap() +} + +#[test] +fn update_and_check_refusals_need_no_server() { + let home = tempdir("refusals"); + home::open(&home).unwrap(); + ok(&home, "machine.list", &[]); + // a record written while machines were attached one by one: native + // attached with a check, docker with a connection of its own + write_record( + &paths::store_record(&home, "pg"), + &SdkValue::map([ + ("name", t("pg")), + ("kind", t("postgresql")), + ("url", t("postgresql://alice@127.0.0.1:1/app")), + ("schema", t("layerfs")), + ("synchronous_commit", t("off")), + ("binding", SdkValue::Bytes(vec![7; 32])), + ("cursor", SdkValue::Bytes(vec![9; 32])), + ( + "machines", + SdkValue::map([ + ( + "native", + SdkValue::map([("checked", SdkValue::map([("at", SdkValue::Int(1))]))]), + ), + ( + "docker", + SdkValue::map([("url", t("postgresql://bob@127.0.0.1:1/app"))]), + ), + ]), + ), + ]), + ) + .unwrap(); + write_record( + &paths::project_record(&home, "pg", "app"), + &SdkValue::map([("name", t("app"))]), + ) + .unwrap(); + write_record( + &paths::store_record(&home, "shop"), + &SdkValue::map([("name", t("shop")), ("kind", t("sqlite"))]), + ) + .unwrap(); + // only a machine with its own connection shows, without the old check + let shown = ok(&home, "store.get", &[("name", t("pg"))]); + assert_eq!(own(&shown), ["docker"]); + let docker = shown.get("machines").and_then(|m| m.get("docker")).unwrap(); + assert_eq!( + docker.get_text("url"), + Some("postgresql://bob@127.0.0.1:1/app") + ); + assert!(docker.get("checked").is_none(), "{shown:?}"); + let native = ok(&home, "machine.get", &[("name", t("native"))]); + assert!(native.get("postgresql").is_none(), "{native:?}"); + let before = record_text(&home, "pg"); + + let update = |args: &[(&str, SdkValue)]| call(&home, "store.update", args).unwrap_err(); + let on = |machine: &str| [("name", t("pg")), ("machine", t(machine))]; + let (code, message) = update(&[("name", t("nope")), ("url", t("postgresql://a@h/d"))]); + assert_eq!(code, "NotFound", "{message}"); + let (code, message) = update(&[("name", t("shop")), ("machine", t("native"))]); + assert_eq!(code, "InvalidInput", "{message}"); + assert!(message.contains("sqlite Store"), "{message}"); + let (code, message) = update(&[("name", t("pg"))]); + assert_eq!(code, "InvalidInput", "{message}"); + assert!(message.starts_with("nothing to update"), "{message}"); + let (code, message) = update(&on("native")); + assert_eq!(code, "InvalidInput", "{message}"); + assert!(message.starts_with("nothing to update"), "{message}"); + let (code, message) = update(&[("name", t("pg")), ("clear", SdkValue::Bool(true))]); + assert_eq!(code, "InvalidInput", "{message}"); + assert!(message.contains("--machine"), "{message}"); + let mut args = on("native").to_vec(); + args.push(("synchronous_commit", t("off"))); + let (code, message) = update(&args); + assert_eq!(code, "InvalidInput", "{message}"); + assert!(message.contains("the Store's setting"), "{message}"); + let mut args = on("docker").to_vec(); + args.extend([ + ("clear", SdkValue::Bool(true)), + ("url", t("postgresql://a@h/d")), + ]); + let (code, message) = update(&args); + assert_eq!(code, "InvalidInput", "{message}"); + let mut args = on("native").to_vec(); + args.push(("clear", SdkValue::Bool(true))); + let (code, message) = update(&args); + assert_eq!(code, "InvalidInput", "{message}"); + assert_eq!( + message, + "machine native has no connection of its own to Store pg" + ); + let mut args = on("nowhere").to_vec(); + args.push(("url", t("postgresql://alice@127.0.0.1:1/app"))); + let (code, message) = update(&args); + assert_eq!(code, "NotFound", "{message}"); + for machine in [None, Some("native")] { + let mut args = vec![("name", t("pg"))]; + args.extend(machine.map(|m| ("machine", t(m)))); + args.push(("url", t("postgresql://alice:secret@127.0.0.1:1/app"))); + let (code, message) = update(&args); + assert_eq!(code, "InvalidInput", "{message}"); + assert!(!message.contains("secret"), "{message}"); + args.pop(); + args.push(("password", t("env:LAYERFS_TEST_UPDATE_UNSET_PASSWORD"))); + let (code, message) = update(&args); + // the reference is resolved before any connection, as connect's is + assert!( + message.contains("LAYERFS_TEST_UPDATE_UNSET_PASSWORD is not set"), + "{code}: {message}" + ); + } + let (code, message) = update(&[("name", t("pg")), ("synchronous_commit", t("sometimes"))]); + assert_eq!(code, "InvalidInput", "{message}"); + // the check from native finds no server: nothing is saved + let mut args = on("native").to_vec(); + args.push(("url", t("postgresql://alice@127.0.0.1:1/app"))); + let (code, message) = update(&args); + assert_eq!(code, "Unavailable", "{message}"); + assert!( + message + .starts_with("machine native cannot reach PostgreSQL Store pg's server at 127.0.0.1: "), + "{message}" + ); + // nor does the Store's own new URL reach one + let (code, message) = update(&[ + ("name", t("pg")), + ("url", t("postgresql://alice@127.0.0.1:1/other")), + ]); + assert_eq!(code, "Unavailable", "{message}"); + assert!(message.starts_with("PostgreSQL Store pg: "), "{message}"); + assert_eq!( + record_text(&home, "pg"), + before, + "a refused update writes nothing" + ); + + // the check on request: a successful answer that says it failed, and + // records nothing + let status = ok(&home, "store.status", &on("native")); + let check = status.get("check").expect("check"); + assert_eq!(check.get_text("machine"), Some("native")); + assert_eq!(check.get_text("host"), Some("127.0.0.1")); + assert_eq!(check.get("ok"), Some(&SdkValue::Bool(false))); + let error = check.get("error").expect("error"); + assert_eq!(error.get_text("code"), Some("Unavailable")); + assert!( + error + .get_text("message") + .is_some_and(|m| m.starts_with("machine native cannot reach PostgreSQL Store pg")), + "{error:?}" + ); + let plain = ok(&home, "store.status", &[("name", t("pg"))]); + assert!(plain.get("check").is_none(), "{plain:?}"); + let refusal = |args: &[(&str, SdkValue)]| call(&home, "store.status", args).unwrap_err(); + let (code, message) = refusal(&[("name", t("shop")), ("machine", t("native"))]); + assert_eq!(code, "InvalidInput", "{message}"); + let (code, message) = refusal(&on("nowhere")); + assert_eq!(code, "NotFound", "{message}"); + assert_eq!(record_text(&home, "pg"), before, "a check writes nothing"); + + // synchronous_commit alone dials nothing + let shown = ok( + &home, + "store.update", + &[("name", t("pg")), ("synchronous_commit", t("local"))], + ); + assert_eq!(shown.get_text("synchronous_commit"), Some("local")); + assert_eq!(own(&shown), ["docker"]); + // clearing a machine's own connection dials nothing either, and the + // record keeps no `machines` once none has one + let mut args = on("docker").to_vec(); + args.push(("clear", SdkValue::Bool(true))); + let shown = ok(&home, "store.update", &args); + assert!(own(&shown).is_empty(), "{shown:?}"); + assert!(!record_text(&home, "pg").contains("machines")); + + // a Sandbox needs no attachment: the create goes on to its reach check, + // which finds no server + let (code, message) = call( + &home, + "sandbox.create", + &[("project", t("pg/app")), ("name", t("t"))], + ) + .unwrap_err(); + assert!(!message.contains("not attached"), "{code}: {message}"); + assert!(!paths::sandbox_dir(&home, "t").exists()); + let _ = std::fs::remove_dir_all(home); +} + +#[test] +fn a_machine_s_own_connection_is_checked_before_it_is_saved() { + let (Ok(url), Ok(_)) = ( + std::env::var("LAYERFS_TEST_PG_URL"), + std::env::var("LAYERFS_TEST_PG_PASSWORD"), + ) else { + eprintln!("NOT_RUN: LAYERFS_TEST_PG_URL / LAYERFS_TEST_PG_PASSWORD unset"); + return; + }; + let home = tempdir("live"); + home::open(&home).unwrap(); + let connected = ok( + &home, + "store.connect", + &[ + ("name", t("pg")), + ("url", t(&url)), + ("password", t("env:LAYERFS_TEST_PG_PASSWORD")), + ], + ); + // connect records no machine: every machine dials the Store's connection + assert!(connected.get("machines").is_none(), "{connected:?}"); + assert!(!record_text(&home, "pg").contains("machines")); + let native = [("name", t("pg")), ("machine", t("native"))]; + // the check on request passes and records nothing + let before = record_text(&home, "pg"); + let status = ok(&home, "store.status", &native); + let check = status.get("check").expect("check"); + assert_eq!(check.get("ok"), Some(&SdkValue::Bool(true)), "{check:?}"); + assert!(check.get_text("server_version").is_some(), "{check:?}"); + assert_eq!(record_text(&home, "pg"), before); + // native with a connection of its own: the same server, the password + // by reference; the record holds no password + let mut args = native.to_vec(); + args.extend([ + ("url", t(&url)), + ("password", t("env:LAYERFS_TEST_PG_PASSWORD")), + ]); + let shown = ok(&home, "store.update", &args); + let own_native = shown.get("machines").and_then(|m| m.get("native")).unwrap(); + assert_eq!(own_native.get_text("url"), Some(url.as_str())); + assert_eq!( + own_native.get_text("password"), + Some("env:LAYERFS_TEST_PG_PASSWORD") + ); + assert!(own_native.get("checked").is_none(), "{shown:?}"); + let secret = std::env::var("LAYERFS_TEST_PG_PASSWORD").unwrap(); + assert!(!record_text(&home, "pg").contains(&secret)); + // another database is not this Store: refused, nothing saved, for the + // machine's connection and for the Store's own + let before = record_text(&home, "pg"); + let other = match url.rsplit_once('/') { + Some((server, _)) => format!("{server}/postgres"), + None => url.clone(), + }; + let mut args = native.to_vec(); + args.push(("url", t(&other))); + let (code, message) = call(&home, "store.update", &args).unwrap_err(); + assert_eq!(code, "Unavailable", "{message}"); + assert!(message.contains("holds another Store than pg"), "{message}"); + // the server answered: no "cannot reach" and no reach hint + assert!(!message.contains("cannot reach"), "{message}"); + let (code, message) = call( + &home, + "store.update", + &[("name", t("pg")), ("url", t(&other))], + ) + .unwrap_err(); + assert!( + matches!(code.as_str(), "Unavailable" | "Failed"), + "{code}: {message}" + ); + assert_eq!(record_text(&home, "pg"), before); + // a role that may not write is refused naming what it lacks + if let (Ok(limited), Ok(_)) = ( + std::env::var("LAYERFS_TEST_PG_LIMITED_URL"), + std::env::var("LAYERFS_TEST_PG_LIMITED_PASSWORD"), + ) { + let mut args = native.to_vec(); + args.extend([ + ("url", t(&limited)), + ("password", t("env:LAYERFS_TEST_PG_LIMITED_PASSWORD")), + ]); + let (code, message) = call(&home, "store.update", &args).unwrap_err(); + assert_eq!(code, "Denied", "{message}"); + assert!(message.contains(" lacks "), "{message}"); + assert_eq!(record_text(&home, "pg"), before); + } else { + eprintln!("NOT_RUN (privileges): LAYERFS_TEST_PG_LIMITED_URL unset"); + } + // the Store's own connection, to the same database again: saved, and + // its calls still open it + let shown = ok( + &home, + "store.update", + &[ + ("name", t("pg")), + ("url", t(&url)), + ("synchronous_commit", t("off")), + ], + ); + assert_eq!(shown.get_text("synchronous_commit"), Some("off")); + ok(&home, "store.stats", &[("name", t("pg"))]); + let mut args = native.to_vec(); + args.push(("clear", SdkValue::Bool(true))); + let shown = ok(&home, "store.update", &args); + assert!(own(&shown).is_empty(), "{shown:?}"); + ok(&home, "store.disconnect", &[("name", t("pg"))]); + let _ = std::fs::remove_dir_all(home); +} diff --git a/api/crates/layerfs-api/tests/sandbox_machines.rs b/api/crates/layerfs-api/tests/sandbox_machines.rs index 837ea54ec2..86d751eba8 100644 --- a/api/crates/layerfs-api/tests/sandbox_machines.rs +++ b/api/crates/layerfs-api/tests/sandbox_machines.rs @@ -128,21 +128,14 @@ fn a_sandbox_runs_where_its_store_can_be_reached() { let home = tempdir("refusals"); home::open(&home).unwrap(); ok(&home, "machine.list", &[]); - // a PostgreSQL Store attached to native and dev, and a sqlite Store on - // native, written as records: the refusals come before any connection + // a PostgreSQL Store and a sqlite Store on native, written as records: + // the refusals come before any connection home::write_record( &paths::store_record(&home, "pg"), &SdkValue::map([ ("name", t("pg")), ("kind", t("postgresql")), ("url", t("postgres://u@127.0.0.1:1/db")), - ( - "machines", - SdkValue::map([ - ("native", SdkValue::Map(Vec::new())), - ("dev", SdkValue::Map(Vec::new())), - ]), - ), ]), ) .unwrap(); @@ -202,22 +195,6 @@ fn a_sandbox_runs_where_its_store_can_be_reached() { assert!(message.ends_with(reason), "{message}"); } } - // a machine the PostgreSQL Store is not attached to is refused naming - // the attach, before any reach or startup - if layerfs_platform::sandbox_allowed( - layerfs_platform::Platform::Docker, - layerfs_platform::HostOs::current(), - layerfs_platform::StoreKind::Postgresql, - ) - .is_ok() - { - let (code, message) = create("docker"); - assert_eq!(code, "Unsupported", "{message}"); - assert_eq!( - message, - "PostgreSQL Store pg is not attached to machine docker; `layerfs store attach pg docker`" - ); - } // an ssh machine left by an older home: its platform was removed in // 2.0, refused before any reach or startup let (code, message) = create("box"); @@ -377,13 +354,8 @@ fn a_postgresql_store_s_sandboxes_on_native_and_docker() { let outcome = std::panic::catch_unwind(|| { // only `machine.start` starts a machine (D11) ok(&home, "machine.start", &[("name", t("docker"))]); - // docker runs the Store's Sandboxes once it is attached, its - // connection checked from the worker container - ok( - &home, - "store.attach", - &[("name", t("pg")), ("machine", t("docker"))], - ); + // docker runs the Store's Sandboxes with no step before the + // create, whose reach check dials from the worker container ok( &home, "sandbox.create", diff --git a/api/mcp/test/profiles.test.mjs b/api/mcp/test/profiles.test.mjs index de0a8c02e9..34c056cc96 100644 --- a/api/mcp/test/profiles.test.mjs +++ b/api/mcp/test/profiles.test.mjs @@ -14,7 +14,7 @@ const nameOf = (spec) => `${spec.family}.${spec.call}`; test("every call is in exactly one bucket, and admin adds the work bucket", () => { assert.deepEqual(check(table), []); - assert.equal(table.length, 85); + assert.equal(table.length, 84); const buckets = table.map(bucketOf); assert.equal(buckets.filter((bucket) => bucket === "work").length, 38); assert.equal(buckets.filter((bucket) => bucket === "admin").length, 40); @@ -32,7 +32,7 @@ test("every call is in exactly one bucket, and admin adds the work bucket", () = const work = tools(table, "work").map(nameOf); const admin = tools(table, "admin").map(nameOf); assert.equal(work.length, 38); - assert.equal(admin.length, 78); + assert.equal(admin.length, 77); for (const name of work) assert.ok(admin.includes(name), `${name} is in work but not admin`); assert.ok(!admin.some((name) => name.startsWith("doctor.")), "a doctor call is a tool"); assert.throws(() => tools(table, "root"), /unknown profile/); diff --git a/api/node/README.md b/api/node/README.md index 8e666010bd..c923cfc3db 100644 --- a/api/node/README.md +++ b/api/node/README.md @@ -2,7 +2,7 @@ The LayerFS SDK for Node: twelve families — `home`, `project`, `branch`, `commit`, `file`, `sandbox`, `workspace`, `machine`, `store`, `backup`, `logs`, `doctor` — over one call -table of 85 calls into one native library. Each function makes one call; it +table of 84 calls into one native library. Each function makes one call; it does not retry, replay or commit on its own. Results are plain objects with snake_case keys, whole numbers are exact (`bigint` beyond `Number.MAX_SAFE_INTEGER`), bytes are `Buffer`s, and the TypeScript typings diff --git a/api/node/src/index.d.ts b/api/node/src/index.d.ts index d404dabcfa..b77b0b8aee 100644 --- a/api/node/src/index.d.ts +++ b/api/node/src/index.d.ts @@ -388,16 +388,24 @@ export interface MachineUpdateOptions extends CallOptions { daemon?: string; } +/** Options of `store.status`; an unknown option is refused. */ +export interface StoreStatusOptions extends CallOptions { + machine?: string; +} + /** Options of `store.connect`; an unknown option is refused. */ export interface StoreConnectOptions extends CallOptions { password?: string; synchronous_commit?: string; } -/** Options of `store.attach`; an unknown option is refused. */ -export interface StoreAttachOptions extends CallOptions { +/** Options of `store.update`; an unknown option is refused. */ +export interface StoreUpdateOptions extends CallOptions { + machine?: string; url?: string; password?: string; + synchronous_commit?: string; + clear?: boolean; } /** Options of `store.push`; an unknown option is refused. */ @@ -622,18 +630,16 @@ export interface StoreFamily { list(options?: CallOptions): Promise; /** a Store's record */ get(name: string, options?: CallOptions): Promise; - /** a Store's live state */ - status(name: string, options?: CallOptions): Promise; + /** a Store's live state; with a machine, whether it reaches a PostgreSQL Store */ + status(name: string, options?: StoreStatusOptions): Promise; /** Store-wide publication and allocation counters */ stats(name: string, options?: CallOptions): Promise; /** an empty sqlite Store on native */ create(name: string, options?: CallOptions): Promise; - /** attach a PostgreSQL database as a Store */ + /** connect a PostgreSQL database as a Store */ connect(name: string, url: string, options?: StoreConnectOptions): Promise; - /** let a machine run Sandboxes of a PostgreSQL Store, after checking it reaches the server */ - attach(name: string, machine: string, options?: StoreAttachOptions): Promise; - /** stop a machine running Sandboxes of a PostgreSQL Store */ - detach(name: string, machine: string, options?: CallOptions): Promise; + /** change a PostgreSQL Store's connection, or the one a machine alone dials */ + update(name: string, options?: StoreUpdateOptions): Promise; /** send a sqlite Store from native to a machine */ push(name: string, machine: string, options?: StorePushOptions): Promise; /** bring a sqlite Store from its machine to native */ diff --git a/api/node/test/branch-file/inventory.mjs b/api/node/test/branch-file/inventory.mjs index 90897de44c..b7ef1c4315 100644 --- a/api/node/test/branch-file/inventory.mjs +++ b/api/node/test/branch-file/inventory.mjs @@ -11,8 +11,8 @@ const root = fileURLToPath(new URL("../../../../", import.meta.url)); const command = process.env.LAYERFS_CLI ?? join(root, "target/debug/layerfs"); const docs = join(root, "docs/architecture/api/sdk"); -test("the SDK pages name all 85 current calls, one `### family.call` heading each", () => { - assert.equal(table.length, 85); +test("the SDK pages name all 84 current calls, one `### family.call` heading each", () => { + assert.equal(table.length, 84); const families = [...new Set(table.map((spec) => spec.family))]; assert.equal(families.length, 12); const index = readFileSync(join(docs, "README.md"), "utf8"); diff --git a/api/node/test/calls.mjs b/api/node/test/calls.mjs index e208e041de..464fbe6859 100644 --- a/api/node/test/calls.mjs +++ b/api/node/test/calls.mjs @@ -14,9 +14,9 @@ const temp = (tag) => { return dir; }; -test("the table: twelve families, 85 calls, each a function", () => { +test("the table: twelve families, 84 calls, each a function", () => { assert.deepEqual(families, ["home", "project", "branch", "commit", "file", "sandbox", "workspace", "machine", "store", "backup", "logs", "doctor"]); - assert.equal(table.length, 85); + assert.equal(table.length, 84); for (const spec of table) assert.equal(typeof layerfs[spec.family][spec.call], "function", `${spec.family}.${spec.call}`); }); diff --git a/api/python/src/layerfs/_client.pyi b/api/python/src/layerfs/_client.pyi index 07dc1e7d60..84e3ad46ef 100644 --- a/api/python/src/layerfs/_client.pyi +++ b/api/python/src/layerfs/_client.pyi @@ -533,8 +533,8 @@ class StoreFamily: """every Store, from the home's records""" def get(self, name: str, *, log_tag: str | None = None) -> Result: """a Store's record""" - def status(self, name: str, *, log_tag: str | None = None) -> Result: - """a Store's live state""" + def status(self, name: str, *, machine: str | None = None, log_tag: str | None = None) -> Result: + """a Store's live state; with a machine, whether it reaches a PostgreSQL Store""" def stats(self, name: str, *, log_tag: str | None = None) -> Result: """Store-wide publication and allocation counters""" def create(self, name: str, *, log_tag: str | None = None) -> Result: @@ -548,19 +548,19 @@ class StoreFamily: synchronous_commit: str | None = None, log_tag: str | None = None, ) -> Result: - """attach a PostgreSQL database as a Store""" - def attach( + """connect a PostgreSQL database as a Store""" + def update( self, name: str, - machine: str, *, + machine: str | None = None, url: str | None = None, password: str | None = None, + synchronous_commit: str | None = None, + clear: bool | None = None, log_tag: str | None = None, ) -> Result: - """let a machine run Sandboxes of a PostgreSQL Store, after checking it reaches the server""" - def detach(self, name: str, machine: str, *, log_tag: str | None = None) -> Result: - """stop a machine running Sandboxes of a PostgreSQL Store""" + """change a PostgreSQL Store's connection, or the one a machine alone dials""" def push( self, name: str, diff --git a/api/python/src/layerfs/aio.pyi b/api/python/src/layerfs/aio.pyi index 2e5dbc6fd5..f5b7060c01 100644 --- a/api/python/src/layerfs/aio.pyi +++ b/api/python/src/layerfs/aio.pyi @@ -538,8 +538,8 @@ class AsyncStoreFamily: """every Store, from the home's records""" async def get(self, name: str, *, log_tag: str | None = None) -> Result: """a Store's record""" - async def status(self, name: str, *, log_tag: str | None = None) -> Result: - """a Store's live state""" + async def status(self, name: str, *, machine: str | None = None, log_tag: str | None = None) -> Result: + """a Store's live state; with a machine, whether it reaches a PostgreSQL Store""" async def stats(self, name: str, *, log_tag: str | None = None) -> Result: """Store-wide publication and allocation counters""" async def create(self, name: str, *, log_tag: str | None = None) -> Result: @@ -553,19 +553,19 @@ class AsyncStoreFamily: synchronous_commit: str | None = None, log_tag: str | None = None, ) -> Result: - """attach a PostgreSQL database as a Store""" - async def attach( + """connect a PostgreSQL database as a Store""" + async def update( self, name: str, - machine: str, *, + machine: str | None = None, url: str | None = None, password: str | None = None, + synchronous_commit: str | None = None, + clear: bool | None = None, log_tag: str | None = None, ) -> Result: - """let a machine run Sandboxes of a PostgreSQL Store, after checking it reaches the server""" - async def detach(self, name: str, machine: str, *, log_tag: str | None = None) -> Result: - """stop a machine running Sandboxes of a PostgreSQL Store""" + """change a PostgreSQL Store's connection, or the one a machine alone dials""" async def push( self, name: str, diff --git a/api/python/test/test_calls.py b/api/python/test/test_calls.py index a5670d1b3f..2e7863fde4 100644 --- a/api/python/test/test_calls.py +++ b/api/python/test/test_calls.py @@ -350,7 +350,7 @@ def test_the_table(self): layerfs.FAMILIES, ["home", "project", "branch", "commit", "file", "sandbox", "workspace", "machine", "store", "backup", "logs", "doctor"], ) - self.assertEqual(len(layerfs.TABLE), 85) + self.assertEqual(len(layerfs.TABLE), 84) for spec in layerfs.TABLE: self.assertTrue(callable(getattr(getattr(layerfs, spec["family"]), spec["call"]))) # the options the Publish change added are in the table the Python diff --git a/control/crates/layerfs-control/src/inventory.rs b/control/crates/layerfs-control/src/inventory.rs index 9e88a7097b..c7a4fa0ab1 100644 --- a/control/crates/layerfs-control/src/inventory.rs +++ b/control/crates/layerfs-control/src/inventory.rs @@ -51,37 +51,6 @@ pub fn stores_on(home: &Path, machine: &str) -> Result> { Ok(on) } -/// The PostgreSQL Stores attached to machine `machine`, sorted by name. -pub fn postgresql_on(home: &Path, machine: &str) -> Result> { - let mut on = Vec::new(); - for store in stores(home)? { - if let StoreRef::Postgres { .. } = store.store_ref()? { - if store.attachment(machine).is_some() { - on.push(store.name().to_owned()); - } - } - } - Ok(on) -} - -/// `store.detach`: refused `InUse` while a Sandbox of Store `store` is -/// recorded on machine `machine`; `detail.related` lists them. -pub fn store_idle_on(home: &Path, store: &str, machine: &str) -> Result<()> { - let on = sandboxes(home)?.into_iter().filter(|sandbox| { - let project = sandbox.project().unwrap_or(""); - sandbox.machine() == machine && project.split_once('/').map(|(s, _)| s) == Some(store) - }); - let names: Vec = on.map(|sandbox| sandbox.name().to_owned()).collect(); - if names.is_empty() { - return Ok(()); - } - let message = format!( - "Store {store} has Sandboxes on machine {machine}: {}; close them first", - names.join(", ") - ); - Err(in_use(message, related("sandbox", &names))) -} - /// The Sandboxes whose record names machine `machine`. pub fn sandboxes_on(home: &Path, machine: &str) -> Result> { let on = sandboxes(home)?.into_iter(); @@ -115,16 +84,13 @@ pub fn sandboxes_of_project(home: &Path, project: &str) -> Result> { Ok(of.map(|sandbox| sandbox.name().to_owned()).collect()) } -/// `machine.remove`: refused `InUse` while the machine holds Stores (or -/// PostgreSQL Stores are attached to it), +/// `machine.remove`: refused `InUse` while the machine holds Stores, /// Sandboxes or old Firecracker records it hosted (removing it would delete /// their microVMs' disks; forgetting those records first lets it go); -/// `detail.related` lists them (#536 §10b). +/// `detail.related` lists them (#536 §10b). A PostgreSQL Store is on no +/// machine and holds none. pub fn can_remove_machine(home: &Path, name: &str) -> Result<()> { - // a PostgreSQL Store attached to it is detached first, as a sqlite - // Store on it is pulled first - let mut stores = stores_on(home, name)?; - stores.extend(postgresql_on(home, name)?); + let stores = stores_on(home, name)?; let sandboxes = sandboxes_on(home, name)?; let hosted = hosted_on(home, name)?; if stores.is_empty() && sandboxes.is_empty() && hosted.is_empty() { diff --git a/control/crates/layerfs-control/src/machine.rs b/control/crates/layerfs-control/src/machine.rs index abe7e9689b..2b571d8103 100644 --- a/control/crates/layerfs-control/src/machine.rs +++ b/control/crates/layerfs-control/src/machine.rs @@ -12,7 +12,6 @@ use std::path::Path; /// What a machine holds, read from the Store and Sandbox records. pub const RELATED: Related = Related { stores_on: inventory::stores_on, - postgresql_on: inventory::postgresql_on, sandboxes_on: inventory::sandboxes_on, can_remove: inventory::can_remove_machine, can_stop: inventory::can_stop_machine, @@ -54,9 +53,21 @@ pub fn create(home: &Path, events: &mut Events<'_>, request: &MachineCreate) -> layerfs_machine::create(home, events, request, RELATED) } -/// `machine.remove`. +/// `machine.remove`. A removed machine's own connection to each +/// PostgreSQL Store is forgotten with it, so a later machine of its name +/// dials the Store's. pub fn remove(home: &Path, name: &MachineName) -> Outcome { - layerfs_machine::remove(home, name, RELATED) + let removed = layerfs_machine::remove(home, name, RELATED)?; + layerfs_store::postgresql::check::forget_machine(home, name).map_err(|refusal| { + layerfs_home::outcome::refused( + &refusal.code, + format!( + "machine {name} was removed, but its own PostgreSQL connection was not forgotten: {}", + refusal.message + ), + ) + })?; + Ok(removed) } /// `machine.update`. diff --git a/control/crates/layerfs-control/src/store.rs b/control/crates/layerfs-control/src/store.rs index a193ee2147..2db10ad376 100644 --- a/control/crates/layerfs-control/src/store.rs +++ b/control/crates/layerfs-control/src/store.rs @@ -1,8 +1,9 @@ //! The Store calls whose checks read who uses a Store (layerfs-project#105): //! each runs `layerfs-store`'s call with the checks from //! [`crate::inventory`]. The Store calls proper (`branch`, `commit`, `file`) -//! and the rest are `layerfs-store`'s own. `store.attach` also reads the -//! machine and reaches it as a Sandbox call would. +//! and the rest are `layerfs-store`'s own. `store.update --machine` and +//! `store.status --machine` also read the machine and reach it as a Sandbox +//! call would. use crate::inventory; use layerfs_bridge::control::{SdkRefusal, SdkValue}; use layerfs_home::events::Events; @@ -10,8 +11,8 @@ use layerfs_home::names::{MachineName, StoreName}; use layerfs_home::outcome::{refused, Outcome}; use layerfs_machine::worker::Unreached; use layerfs_store::place::{self, Executor}; -use layerfs_store::postgresql::{attach, connect}; -use layerfs_store::record::{Attachment, StoreKind}; +use layerfs_store::postgresql::{check, connect, update as store_update}; +use layerfs_store::record::{OwnConnection, StoreKind, StoreRecord}; use layerfs_store::sqlite::family::StorePush; use layerfs_store::Related; use std::path::Path; @@ -50,18 +51,23 @@ pub fn disconnect(home: &Path, name: &StoreName) -> Outcome { layerfs_store::postgresql::connect::disconnect(home, name, RELATED) } -/// `store.attach(name, machine, { url, password })`. No `Debug`: it names a -/// password reference. +/// `store.update(name, { machine, url, password, synchronous_commit, +/// clear })`. No `Debug`: it names a password reference. #[derive(Clone)] -pub struct StoreAttach { +pub struct StoreUpdate { /// The PostgreSQL Store. pub name: StoreName, - /// The machine that may run its Sandboxes. - pub machine: MachineName, - /// The URL the machine dials (no password), when not the Store's. + /// The machine whose own connection is set or cleared; none for the + /// Store's own. + pub machine: Option, + /// The new URL (no password). pub url: Option, - /// The password's reference, when not the Store's. + /// The password's new reference. pub password: Option, + /// The Store's new `synchronous_commit`. + pub synchronous_commit: Option, + /// Remove the machine's own connection. + pub clear: bool, } /// Refused unless machine `machine` exists and its platform runs @@ -107,10 +113,10 @@ fn from_machine( "InvalidInput" if message.contains("no operation store.reach") => refused( "Unsupported", format!( - "machine {machine}'s worker predates store.attach; `layerfs machine stop {machine}` and `layerfs machine start {machine}` update it" + "machine {machine}'s worker predates this check; `layerfs machine stop {machine}` and `layerfs machine start {machine}` update it" ), ), - "Unavailable" if !message.starts_with(attach::ANOTHER_STORE) => { + "Unavailable" if !message.starts_with(check::ANOTHER_STORE) => { let hint = hint.map(|hint| format!("; {hint}")).unwrap_or_default(); refused( "Unavailable", @@ -126,66 +132,188 @@ fn from_machine( } } -/// `store.attach`: checks machine `machine`'s connection to PostgreSQL -/// Store `name` from that machine (in this process, or in its worker), then -/// records the attachment; on a machine already attached it checks the new -/// connection and replaces the old one. Every refusal of the input comes -/// before any connection; a failed check records nothing. Answers the -/// Store as `store.get` shows it. -pub fn attach(home: &Path, events: &mut Events<'_>, request: &StoreAttach) -> Outcome { - let (name, machine) = (&request.name, request.machine.as_str()); - let record = attach::postgresql_record(home, name)?; - runs_postgresql_sandboxes(home, name, machine)?; - if let Some(url) = &request.url { - connect::refuse_password(url)?; - connect::target(url)?; - } - if let Some(password) = &request.password { - layerfs_home::secrets::resolve(password)?; - } - let attachment = Attachment { - url: request.url.clone(), - password: request.password.clone(), - checked: None, +/// The check of machine `machine`'s connection to PostgreSQL Store +/// `record` (its own connection where the record names one, else the +/// Store's), run from that machine: in this process, or in its worker as +/// `store.reach`. The URL is built as the machine's Sandboxes' calls build +/// it. Answers the check, `{ at, server_version, latency_ms }`, or its +/// refusal; either way with the host it dialled, once known. +fn check_from( + home: &Path, + events: &mut Events<'_>, + record: &StoreRecord, + machine: &str, +) -> (Option, Result) { + let name = record.name(); + let (postgresql, hint) = match layerfs_sandbox::route::reached(home, machine, record) { + Ok(reached) => reached, + Err(refusal) => return (None, Err(refusal)), + }; + let host = match postgresql.url().and_then(connect::target) { + Ok((host, _, _)) => host, + Err(refusal) => return (None, Err(refusal)), }; - // the URL the machine dials, built as its Sandboxes' calls build it - let trial = record.clone().with_attachment(machine, &attachment); - let (postgresql, hint) = layerfs_sandbox::route::reached(home, machine, &trial)?; - let (host, _, _) = connect::target(postgresql.url()?)?; - let checked = match place::on_machine(home, machine)? { - Executor::Local => attach::check(&record, &postgresql), + let executor = match place::on_machine(home, machine) { + Ok(executor) => executor, + Err(refusal) => return (Some(host), Err(refusal)), + }; + let checked = match executor { + Executor::Local => check::check(record, &postgresql), Executor::Worker(mut worker) => { let request = SdkValue::map([ - ("store", SdkValue::text(name.as_str())), + ("store", SdkValue::text(name)), ("store_record", record.to_record().clone()), ("postgresql", postgresql.encode()), ]); let mut sink = |event| events.event(event); match worker.attempt("store.reach", request, &mut sink) { Ok(outcome) => outcome, - Err(Unreached(refusal)) => Err(refusal), + // the machine's worker was not reached: its own refusal, + // which names the start, not a server the check never dialled + Err(Unreached(refusal)) => return (Some(host), Err(refusal)), } } + }; + let checked = checked.map_err(|refusal| from_machine(refusal, (name, machine, &host), hint)); + (Some(host), checked) +} + +/// `store.update`: changes PostgreSQL Store `name`'s own connection +/// (`layerfs-store`'s part), or with `machine` the connection that machine +/// alone dials: `url` and `password` replace the ones it has (either may +/// be given alone), checked from that machine before they are saved, and +/// `clear` removes its own connection, dialling nothing. Every refusal of +/// the input comes before any connection; a failed check saves nothing. +/// Answers the Store as `store.get` shows it. +pub fn update(home: &Path, events: &mut Events<'_>, request: &StoreUpdate) -> Outcome { + let name = &request.name; + let Some(machine) = request.machine.as_ref().map(|m| m.as_str()) else { + if request.clear { + return Err(refused( + "InvalidInput", + "--clear removes a machine's own connection; name it with --machine", + )); + } + return store_update::update( + home, + &store_update::StoreUpdate { + name: name.clone(), + url: request.url.clone(), + password: request.password.clone(), + synchronous_commit: request.synchronous_commit.clone(), + }, + ); + }; + let record = check::postgresql_record(home, name)?; + if request.synchronous_commit.is_some() { + return Err(refused( + "InvalidInput", + "synchronous_commit is the Store's setting; update it without --machine", + )); + } + let given = request.url.is_some() || request.password.is_some(); + let current = record.own_connection(machine); + if request.clear { + if given { + return Err(refused( + "InvalidInput", + "--clear takes no --url or --password", + )); + } + if current.is_none() { + return Err(refused( + "InvalidInput", + format!("machine {machine} has no connection of its own to Store {name}"), + )); + } + check::save_own( + home, + name, + machine, + &OwnConnection::default(), + record.binding(), + )?; + return layerfs_store::sqlite::family::get(home, name); + } + if !given { + return Err(refused( + "InvalidInput", + "nothing to update: give --url or --password for the machine, or --clear", + )); + } + runs_postgresql_sandboxes(home, name, machine)?; + if let Some(url) = &request.url { + connect::refuse_password(url)?; + connect::target(url)?; } - .map_err(|refusal| from_machine(refusal, (name, machine, &host), hint))?; - let attachment = Attachment { - checked: Some(checked), - ..attachment + if let Some(password) = &request.password { + layerfs_home::secrets::resolve(password)?; + } + let current = current.unwrap_or_default(); + let own = OwnConnection { + url: request.url.clone().or(current.url), + password: request.password.clone().or(current.password), }; - attach::record(home, name, machine, &attachment, record.binding())?; + let trial = record.clone().with_own_connection(machine, &own); + check_from(home, events, &trial, machine).1?; + check::save_own(home, name, machine, &own, record.binding())?; layerfs_store::sqlite::family::get(home, name) } +/// `store.status`: a Store's live state; with `machine`, also the check of +/// that machine's connection to a PostgreSQL Store, run from the machine +/// now and recorded nowhere: `check: { machine, host, ok: true, at, +/// server_version, latency_ms }`, or `ok: false` with the refusal as +/// `error: { code, message }`. A sqlite Store, an unknown machine or a +/// platform that runs no PostgreSQL Sandboxes refuses the call. +pub fn status( + home: &Path, + events: &mut Events<'_>, + name: &StoreName, + machine: Option<&MachineName>, +) -> Outcome { + let Some(machine) = machine.map(|m| m.as_str()) else { + return layerfs_store::sqlite::family::status(home, name); + }; + let record = check::postgresql_record(home, name)?; + runs_postgresql_sandboxes(home, name, machine)?; + let status = layerfs_store::sqlite::family::status(home, name)?; + let (host, checked) = check_from(home, events, &record, machine); + let mut fields = vec![ + ("machine".to_owned(), SdkValue::text(machine)), + ( + "host".to_owned(), + host.map(SdkValue::Text).unwrap_or_default(), + ), + ("ok".to_owned(), SdkValue::Bool(checked.is_ok())), + ]; + match checked { + Ok(SdkValue::Map(checked)) => fields.extend(checked), + Ok(_) => {} + Err(refusal) => fields.push(( + "error".to_owned(), + SdkValue::map([ + ("code", SdkValue::Text(refusal.code)), + ("message", SdkValue::Text(refusal.message)), + ]), + )), + } + let SdkValue::Map(mut out) = status else { + return Ok(status); + }; + out.push(("check".to_owned(), SdkValue::Map(fields))); + Ok(SdkValue::Map(out)) +} + /// `store.reach` in a machine's worker: `{ store, store_record, postgresql }` /// from the caller, which resolved the password and the URL this machine -/// dials. Answers the check `store.attach` records. +/// dials. Answers the check `store.update --machine` and `store.status +/// --machine` ask for. pub fn reach_in_worker(args: &SdkValue) -> Outcome { let store = args.get_text("store").unwrap_or(""); StoreName::parse(store)?; - let record = layerfs_store::record::StoreRecord::from_record( - store, - args.get("store_record").cloned().unwrap_or_default(), - ); + let record = + StoreRecord::from_record(store, args.get("store_record").cloned().unwrap_or_default()); if record.kind()? != StoreKind::Postgresql { return Err(refused( "InvalidInput", @@ -201,13 +329,5 @@ pub fn reach_in_worker(args: &SdkValue) -> Outcome { "store.reach carries the resolved postgresql value", ) })?; - attach::check(&record, &postgresql) -} - -/// `store.detach`: forgets PostgreSQL Store `name`'s attachment to -/// `machine`; refused `InUse` while a Sandbox of the Store is recorded -/// there. Answers the Store as `store.get` shows it. -pub fn detach(home: &Path, name: &StoreName, machine: &MachineName) -> Outcome { - attach::detach(home, name, machine.as_str(), inventory::store_idle_on)?; - layerfs_store::sqlite::family::get(home, name) + check::check(&record, &postgresql) } diff --git a/control/crates/layerfs-control/src/worker_server/ops.rs b/control/crates/layerfs-control/src/worker_server/ops.rs index 425ded31ca..71fe133ef1 100644 --- a/control/crates/layerfs-control/src/worker_server/ops.rs +++ b/control/crates/layerfs-control/src/worker_server/ops.rs @@ -110,8 +110,9 @@ pub(super) fn run( // removes one leftover under this root, under its guard (#180) "doctor.fix" => crate::doctor::leftover_fix::fix_in_worker(root, &args), "store.diagnose" => crate::doctor::diagnose_store_in_worker(root, args, events), - // `store.attach`'s check of this machine's connection to a - // PostgreSQL Store: the caller sends the record and what it dials + // the check of this machine's connection to a PostgreSQL Store + // (`store.update`, `store.status`): the caller sends the record and + // what it dials "store.reach" => crate::store::reach_in_worker(&args), // one Store call, on the Store file under this worker's root (2.0) "store.call" => { diff --git a/control/crates/layerfs-control/tests/inventory.rs b/control/crates/layerfs-control/tests/inventory.rs index badc1b0282..035e3697af 100644 --- a/control/crates/layerfs-control/tests/inventory.rs +++ b/control/crates/layerfs-control/tests/inventory.rs @@ -142,36 +142,21 @@ fn relationships_are_read_from_the_records() { let _ = std::fs::remove_dir_all(&home); } -/// A PostgreSQL Store is attached where its record's `machines` names (a -/// record without them, to `native` alone); `store.detach` is refused while -/// a Sandbox of the Store is recorded on that machine, and a machine with -/// a PostgreSQL Store attached is not removed. +/// A PostgreSQL Store is on no machine: a machine's own connection to one +/// (its record's `machines`) and a Sandbox of one hold no Store on the +/// machine, so only its sqlite Stores, Sandboxes and hosted records refuse +/// its removal. #[test] -fn postgresql_attachments_are_read_from_the_store_records() { +fn a_postgresql_store_holds_no_machine() { let home = home(); - assert_eq!(inventory::postgresql_on(&home, "native").unwrap(), ["pg"]); - assert!(inventory::postgresql_on(&home, "dev").unwrap().is_empty()); record( &paths::store_record(&home, "team"), r#"{"kind":"postgresql","machines":{"dev":{"url":"postgresql://lfs_dev@10.0.0.5/team"}}}"#, ); - assert_eq!(inventory::postgresql_on(&home, "dev").unwrap(), ["team"]); - assert_eq!(inventory::postgresql_on(&home, "native").unwrap(), ["pg"]); - record( - &paths::sandbox_record(&home, "t"), - r#"{"v":1,"name":"t","project":"team/api","machine":"dev"}"#, - ); - let refusal = inventory::store_idle_on(&home, "team", "dev").unwrap_err(); - assert_eq!(refusal.code, "InUse", "{}", refusal.message); - assert_eq!( - refusal.message, - "Store team has Sandboxes on machine dev: t; close them first" - ); - assert!(inventory::store_idle_on(&home, "team", "native").is_ok()); - assert!(inventory::store_idle_on(&home, "pg", "dev").is_ok()); + assert_eq!(inventory::stores_on(&home, "dev").unwrap(), ["shop"]); let refusal = inventory::can_remove_machine(&home, "dev").unwrap_err(); assert!( - refusal.message.contains("Stores [shop, team]"), + refusal.message.contains("Stores [shop]"), "{}", refusal.message ); diff --git a/control/crates/layerfs-control/tests/typed_records.rs b/control/crates/layerfs-control/tests/typed_records.rs index 0f5b92b5ae..d8568ef8e5 100644 --- a/control/crates/layerfs-control/tests/typed_records.rs +++ b/control/crates/layerfs-control/tests/typed_records.rs @@ -146,7 +146,6 @@ mod machines { let derived = MachineDerived { live: Some(("stopped".into(), Some("its probe did not answer".into()))), stores: vec!["shop".into()], - postgresql: vec!["team".into()], sandboxes: vec!["t1".into()], }; let shown = record.to_sdk(View::List, &derived).unwrap(); @@ -162,7 +161,6 @@ mod machines { "memory", "disk", "stores", - "postgresql", "sandboxes", "image", "error" @@ -173,7 +171,6 @@ mod machines { assert_eq!(shown.get_text("error"), Some("its probe did not answer")); assert_eq!(shown.get("cpus"), Some(&SdkValue::Int(2))); assert_eq!(print(shown.get("stores").unwrap()), r#"["shop"]"#); - assert_eq!(print(shown.get("postgresql").unwrap()), r#"["team"]"#); assert_eq!(print(shown.get("sandboxes").unwrap()), r#"["t1"]"#); assert_eq!(record.state(Some("missing")).unwrap(), "missing"); assert_eq!(record.state(None).unwrap(), "running"); @@ -205,7 +202,6 @@ mod machines { "memory", "disk", "stores", - "postgresql", "sandboxes", "docker" ] diff --git a/control/crates/layerfs-machine/src/list.rs b/control/crates/layerfs-machine/src/list.rs index 15841e5e4f..7667d5d938 100644 --- a/control/crates/layerfs-machine/src/list.rs +++ b/control/crates/layerfs-machine/src/list.rs @@ -83,7 +83,6 @@ pub(super) fn summary( let derived = MachineDerived { live, stores: (related.stores_on)(home, record.name())?, - postgresql: (related.postgresql_on)(home, record.name())?, sandboxes: (related.sandboxes_on)(home, record.name())?, }; record.to_sdk(View::Full, &derived) diff --git a/control/crates/layerfs-machine/src/record.rs b/control/crates/layerfs-machine/src/record.rs index 2b31926797..bac8ba6f49 100644 --- a/control/crates/layerfs-machine/src/record.rs +++ b/control/crates/layerfs-machine/src/record.rs @@ -262,8 +262,6 @@ pub struct MachineDerived { pub live: Option<(String, Option)>, /// The sqlite Stores on it. pub stores: Vec, - /// The PostgreSQL Stores attached to it. - pub postgresql: Vec, /// The Sandboxes recorded on it. pub sandboxes: Vec, } @@ -316,9 +314,8 @@ impl MachineRecord { /// The machine as `machine.get`, each `machine.list` item, `start`, /// `stop`, `update` and `create` show it: name, platform, backend, - /// `supports`, state, size, sqlite Stores, the PostgreSQL Stores - /// attached to it and Sandboxes, then the settings it - /// has (`image`, `docker`, `wsl`, `error`, and a removed platform's + /// `supports`, state, size, sqlite Stores and Sandboxes, then the + /// settings it has (`image`, `docker`, `wsl`, `error`, and a removed platform's /// `ssh`, `host` or `vm_name`). A machine of a removed /// platform has backend `-`, supports nothing, state `unsupported` and /// the removal as its `error`. @@ -364,7 +361,6 @@ impl MachineRecord { ("memory", self.field("memory").cloned().unwrap_or_default()), ("disk", self.field("disk").cloned().unwrap_or_default()), ("stores", list(&derived.stores)), - ("postgresql", list(&derived.postgresql)), ("sandboxes", list(&derived.sandboxes)), ]; for extra in ["image", "os_image", "docker", "wsl", "daemon", "error"] { diff --git a/control/crates/layerfs-machine/src/related.rs b/control/crates/layerfs-machine/src/related.rs index 2b69423021..26ee9a2c79 100644 --- a/control/crates/layerfs-machine/src/related.rs +++ b/control/crates/layerfs-machine/src/related.rs @@ -15,8 +15,6 @@ type Check = fn(home: &Path, name: &str) -> Result<(), SdkRefusal>; pub struct Related { /// The Stores whose record names the machine. pub stores_on: Names, - /// The PostgreSQL Stores attached to the machine. - pub postgresql_on: Names, /// The Sandboxes whose record names the machine. pub sandboxes_on: Names, /// `machine.remove`: refused while the machine diff --git a/control/crates/layerfs-machine/tests/docker_socket_choice.rs b/control/crates/layerfs-machine/tests/docker_socket_choice.rs index 1d87cb7f86..0621044303 100644 --- a/control/crates/layerfs-machine/tests/docker_socket_choice.rs +++ b/control/crates/layerfs-machine/tests/docker_socket_choice.rs @@ -30,7 +30,6 @@ fn free(_: &Path, _: &str) -> Result<(), SdkRefusal> { const RELATED: Related = Related { stores_on: none, - postgresql_on: none, sandboxes_on: none, can_remove: free, can_stop: free, diff --git a/control/crates/layerfs-sandbox/src/create.rs b/control/crates/layerfs-sandbox/src/create.rs index e88c443fb7..41c7d475fb 100644 --- a/control/crates/layerfs-sandbox/src/create.rs +++ b/control/crates/layerfs-sandbox/src/create.rs @@ -38,8 +38,8 @@ pub struct Created { } /// Starts the Sandbox: with no `machine`, on a sqlite Store's machine, or -/// `native` for a PostgreSQL Store (#536 §10), which must be attached to -/// the machine. +/// `native` for a PostgreSQL Store (#536 §10), which runs on any machine +/// whose platform runs PostgreSQL Sandboxes. pub fn create( home: &Path, events: &mut Events<'_>, @@ -102,20 +102,9 @@ pub fn create( if layerfs_machine::record::MachineRecord::read(&home, machine)?.is_none() { return Err(refused("NotFound", format!("no machine named {machine}"))); } - // a PostgreSQL Store is on no machine: its Sandboxes run on the machines - // it is attached to, each dialling the server with its own connection - // or the Store's, and the create's reach check proves it can - if let StoreRef::Postgres { .. } = target.store_ref { - if target.record.attachment(machine).is_none() { - return Err(refused( - "Unsupported", - format!( - "PostgreSQL Store {store} is not attached to machine {machine}; `layerfs store attach {store} {machine}`", - store = target.store - ), - )); - } - } + // a PostgreSQL Store is on no machine: its Sandboxes run on any machine + // the support table allows, each dialling the server with its own + // connection or the Store's, and the create's reach check proves it can if let StoreRef::Sqlite { .. } = target.store_ref { let store_machine = target.store_ref.machine()?; if store_machine != machine { diff --git a/control/crates/layerfs-sandbox/src/route.rs b/control/crates/layerfs-sandbox/src/route.rs index 33799ecdc1..2703ee0060 100644 --- a/control/crates/layerfs-sandbox/src/route.rs +++ b/control/crates/layerfs-sandbox/src/route.rs @@ -89,13 +89,13 @@ enum RunRoot { } /// A PostgreSQL Store's `postgresql` argument for a Sandbox on `machine` -/// (#535 §7.4): the URL the machine's attachment names, else the record's, +/// (#535 §7.4): the URL the machine's own connection names, else the record's, /// with its password, resolved here in the caller, with a loopback host /// rewritten to the machine's alias or refused /// (`layerfs_machine::platform::reach`). It travels only in the one request that runs /// the call and is never written to a record. /// With it, what a failed reach check tells a user to check on that -/// machine's platform, if anything. `store.attach` builds its check's +/// machine's platform, if anything. `store.update` and `store.status` build their check's /// argument here too. pub fn reached( home: &Path, diff --git a/control/crates/layerfs-store/src/postgresql/attach.rs b/control/crates/layerfs-store/src/postgresql/check.rs similarity index 62% rename from control/crates/layerfs-store/src/postgresql/attach.rs rename to control/crates/layerfs-store/src/postgresql/check.rs index 4923ea0caa..a0a21582c9 100644 --- a/control/crates/layerfs-store/src/postgresql/attach.rs +++ b/control/crates/layerfs-store/src/postgresql/check.rs @@ -1,15 +1,15 @@ -//! A PostgreSQL Store's machines (supersedes #536 C4, takes up H9): each -//! machine that may run its Sandboxes is attached in the Store's record, -//! with the Store's connection or one of its own (a URL without a password -//! and a password reference). `store connect` attaches `native`; -//! `store attach` checks a machine's connection from that machine before -//! it records it, and `store detach` forgets it. This module is the -//! Store's part: the check run on the side that dials, and the record -//! writes. Which side dials, and the URL a machine dials, are the caller's -//! (`layerfs-control`, through the Sandbox route). +//! A PostgreSQL Store's connection, checked from the side that dials it. +//! A Store's Sandboxes run on any machine whose platform runs PostgreSQL +//! Sandboxes (supersedes the per-machine attachments of 2026-11-12; #536 +//! C4 again): each dials the Store's connection, or its own where +//! `store.update --machine` gave it one (H9), kept in the Store's record as +//! `machines.`. This module is the Store's part: the check run on +//! the side that dials, and the record writes. Which side dials, and the +//! URL a machine dials, are the caller's (`layerfs-control`, through the +//! Sandbox route). use crate::open_args::PgSelection; use crate::postgresql::connect::SCHEMA; -use crate::record::{Attachment, StoreKind, StoreRecord}; +use crate::record::{OwnConnection, StoreKind, StoreRecord}; use layerfs_bridge::control::{SdkRefusal, SdkValue}; use layerfs_home::home; use layerfs_home::names::StoreName; @@ -18,18 +18,17 @@ use std::path::Path; type Result = std::result::Result; -/// An `InUse` check on Store `store`'s Sandboxes on machine `machine`. -pub type MachineIdle = fn(home: &Path, store: &str, machine: &str) -> Result<()>; - /// The PostgreSQL Store `name`'s record: `NotFound` when there is none, -/// `InvalidInput` naming `store push` for a sqlite Store. +/// `InvalidInput` for a sqlite Store, which has no connection. pub fn postgresql_record(home: &Path, name: &str) -> Result { let record = StoreRecord::of(home, name)?; match record.kind()? { StoreKind::Postgresql => Ok(record), StoreKind::Sqlite => Err(refused( "InvalidInput", - format!("Store {name} is a sqlite Store; `store push {name} ` places it"), + format!( + "Store {name} is a sqlite Store, which has no connection; `store push {name} ` places it" + ), )), } } @@ -92,15 +91,29 @@ pub fn check(record: &StoreRecord, postgresql: &PgSelection) -> Result } } -/// Records Store `name`'s attachment to `machine` once its check passed, -/// under the home lock and the Store's: `Conflict` when the Store was +/// The light check of a connection the Store's own calls use: the server +/// at `postgresql` is reached and its schema holds Store `record`'s +/// binding. A role that may only read passes, as it does `store.connect`. +pub fn same_store(record: &StoreRecord, postgresql: &PgSelection) -> Result<()> { + let binding = crate::postgresql::open::read_binding(postgresql)?; + if record.binding() != Some(binding.as_slice()) { + return Err(refused( + "Unavailable", + format!("{ANOTHER_STORE} {}", record.name()), + )); + } + Ok(()) +} + +/// Saves `machine`'s own connection to Store `name` (an empty one removes +/// it), under the home lock and the Store's: `Conflict` when the Store was /// disconnected, or connected again to another database (`binding` -/// differs), since the check. -pub fn record( +/// differs), since it was read. +pub fn save_own( home_dir: &Path, name: &StoreName, machine: &str, - attachment: &Attachment, + own: &OwnConnection, binding: Option<&[u8]>, ) -> Result<()> { let _lock = home::lock::take(home_dir)?; @@ -109,26 +122,23 @@ pub fn record( let Some(record) = record else { return Err(refused( "Conflict", - format!("Store {name} changed while machine {machine} was checked; attach it again"), + format!("Store {name} changed while machine {machine} was checked; update it again"), )); }; - record.with_attachment(machine, attachment).write(home_dir) + record.with_own_connection(machine, own).write(home_dir) } -/// `store.detach`: forgets Store `name`'s attachment to `machine`, under -/// the home lock and the Store's. `InvalidInput` when it is not attached -/// there; `InUse` (from `idle`) while a Sandbox of the Store is recorded on -/// that machine. Its data and its other machines stay. -pub fn detach(home_dir: &Path, name: &StoreName, machine: &str, idle: MachineIdle) -> Result<()> { +/// Forgets machine `machine`'s own connection in every PostgreSQL Store of +/// the home (its `machine.remove`): a later machine of the same name dials +/// each Store's connection. Under the home lock and each Store's. +pub fn forget_machine(home_dir: &Path, machine: &str) -> Result<()> { let _lock = home::lock::take(home_dir)?; - let _store = crate::record::lock(home_dir, name)?; - let record = postgresql_record(home_dir, name)?; - if record.attachment(machine).is_none() { - return Err(refused( - "InvalidInput", - format!("Store {name} is not attached to machine {machine}"), - )); + for record in crate::record::stores(home_dir)? { + if record.own_connection(machine).is_none() { + continue; + } + let _store = crate::record::lock(home_dir, record.name())?; + record.without_own_connection(machine).write(home_dir)?; } - idle(home_dir, name, machine)?; - record.without_attachment(machine).write(home_dir) + Ok(()) } diff --git a/control/crates/layerfs-store/src/postgresql/connect.rs b/control/crates/layerfs-store/src/postgresql/connect.rs index 08adce740c..857e698c5d 100644 --- a/control/crates/layerfs-store/src/postgresql/connect.rs +++ b/control/crates/layerfs-store/src/postgresql/connect.rs @@ -150,9 +150,22 @@ pub fn target(url: &str) -> Result<(String, String, String)> { Ok((decode(&host).to_ascii_lowercase(), port, database)) } +/// Refuses a `synchronous_commit` that is not one of PostgreSQL's five. +pub fn synchronous_commit(sync: &str) -> Result<()> { + if matches!( + sync, + "off" | "local" | "remote_write" | "on" | "remote_apply" + ) { + return Ok(()); + } + Err(refused( + "InvalidInput", + format!("synchronous_commit is off, local, remote_write, on or remote_apply, not {sync}"), + )) +} + /// Refuses `Exists` when a Store is named `name`, or when another -/// PostgreSQL Store of this home holds the same data: the same server and -/// database by its URL, or, once read, the same schema binding. +/// PostgreSQL Store of this home holds the same data. fn unclaimed( home: &Path, name: &str, @@ -162,7 +175,22 @@ fn unclaimed( if paths::store_record(home, name).exists() { return Err(refused("Exists", format!("a Store named {name} exists"))); } + unclaimed_by_others(home, name, target_of, binding) +} + +/// Refuses `Exists` when a PostgreSQL Store of this home other than `name` +/// holds the same data: the same server and database by its URL, or, once +/// read, the same schema binding. +pub fn unclaimed_by_others( + home: &Path, + name: &str, + target_of: &(String, String, String), + binding: Option<&[u8]>, +) -> Result<()> { for other in paths::names(&home.join("store")).map_err(failed)? { + if other == name { + continue; + } let Some(record) = StoreRecord::read(home, &other)? else { continue; }; @@ -187,7 +215,7 @@ fn unclaimed( Ok(()) } -/// `store.connect`: attach a PostgreSQL database as a Store. Every +/// `store.connect`: record a PostgreSQL database as a Store. Every /// refusal of its input comes before any effect. The server's schema and /// projects are read before the home lock; the lock covers the re-checks /// and the record writes only. @@ -196,17 +224,7 @@ pub fn connect(home: &Path, request: &StoreConnect) -> Outcome { let given = request.url.as_str(); refuse_password(given)?; let sync = request.synchronous_commit.as_deref().unwrap_or("local"); - if !matches!( - sync, - "off" | "local" | "remote_write" | "on" | "remote_apply" - ) { - return Err(refused( - "InvalidInput", - format!( - "synchronous_commit is off, local, remote_write, on or remote_apply, not {sync}" - ), - )); - } + synchronous_commit(sync)?; let password = request.password.as_deref(); if let Some(password) = password { resolve(password)?; @@ -233,15 +251,6 @@ pub fn connect(home: &Path, request: &StoreConnect) -> Outcome { }; let record = record.with("binding", SdkValue::Bytes(binding.clone())); let record = record.with("cursor", SdkValue::Bytes(fresh.cursor_key.to_vec())); - // `native` is attached with the Store's connection: this process dials - // as its Sandboxes do. Its check is recorded when it passes; a role - // that may not write leaves it unchecked (`store attach native` - // checks it again and names what is missing) - let native = crate::record::Attachment { - checked: crate::postgresql::attach::check(&record, &probe).ok(), - ..Default::default() - }; - let record = record.with_attachment("native", &native); let directory = paths::store_dir(home, name); // the projects the schema holds become this home's project records; a // schema that does not open is not connected @@ -272,10 +281,6 @@ pub fn connect(home: &Path, request: &StoreConnect) -> Outcome { ("kind", SdkValue::text("postgresql")), ("schema", SdkValue::text(SCHEMA)), ("synchronous_commit", SdkValue::text(sync)), - ( - "machines", - record.field("machines").cloned().unwrap_or_default(), - ), ("projects", SdkValue::List(projects)), ])) } diff --git a/control/crates/layerfs-store/src/postgresql/mod.rs b/control/crates/layerfs-store/src/postgresql/mod.rs index 3cd01ac804..6f4dc89b02 100644 --- a/control/crates/layerfs-store/src/postgresql/mod.rs +++ b/control/crates/layerfs-store/src/postgresql/mod.rs @@ -1,12 +1,14 @@ -//! PostgreSQL Stores: connect and disconnect, the machines a Store is -//! attached to (`attach`), opening a database session +//! PostgreSQL Stores: connect, update and disconnect, the check of a +//! connection from the side that dials it (`check`), opening a database session //! (`open`, the PostgreSQL URL and Store open), and the held host sessions //! Store calls read through (`reader`). Never imports `sqlite/`. #[doc(hidden)] -pub mod attach; +pub mod check; #[doc(hidden)] pub mod connect; #[doc(hidden)] pub mod open; #[doc(hidden)] pub mod reader; +#[doc(hidden)] +pub mod update; diff --git a/control/crates/layerfs-store/src/postgresql/update.rs b/control/crates/layerfs-store/src/postgresql/update.rs new file mode 100644 index 0000000000..bc4ad87831 --- /dev/null +++ b/control/crates/layerfs-store/src/postgresql/update.rs @@ -0,0 +1,107 @@ +//! `store.update` of a PostgreSQL Store's own connection: its URL (no +//! password), its password's reference and `synchronous_commit`. A new URL +//! or password is checked from this process first (the server is reached +//! and holds this Store), as its Store calls will dial it. A machine's own +//! connection is `layerfs-control`'s part of the call, checked from that +//! machine. +use crate::postgresql::check::{postgresql_record, same_store}; +use crate::postgresql::connect::{ + refuse_password, selection, synchronous_commit, target, unclaimed_by_others, +}; +use crate::record::StoreRecord; +use layerfs_bridge::control::{SdkRefusal, SdkValue}; +use layerfs_home::home; +use layerfs_home::names::StoreName; +use layerfs_home::outcome::{refused, Outcome}; +use layerfs_home::secrets::resolve; +use std::path::Path; + +/// What `store.update` changes of a Store's own connection. No `Debug`: it +/// names a password reference. +#[derive(Clone)] +pub struct StoreUpdate { + /// The PostgreSQL Store. + pub name: StoreName, + /// Its new URL, without a password. + pub url: Option, + /// Its password's new reference. + pub password: Option, + /// Its new `synchronous_commit`. + pub synchronous_commit: Option, +} + +/// `record` with the request's fields in place of its own. +fn changed(record: StoreRecord, request: &StoreUpdate) -> StoreRecord { + let fields = [ + ("url", &request.url), + ("password", &request.password), + ("synchronous_commit", &request.synchronous_commit), + ]; + fields + .into_iter() + .fold(record, |record, (field, value)| match value { + Some(value) => record.with(field, SdkValue::text(value)), + None => record, + }) +} + +/// `store.update` without a machine: changes the Store's URL, password +/// reference and `synchronous_commit`. Every refusal of its input comes +/// before any connection; a new URL or password is dialled once from here +/// and must reach this Store (`Unavailable` otherwise, `Exists` when the +/// URL names a database another Store of the home connects). The record is +/// written under the home lock and the Store's, `Conflict` when it was +/// disconnected or connected to another database meanwhile; this +/// process's open sessions on it are closed (`InUse` while one of its +/// calls runs). A running Sandbox keeps the connection it started with. +pub fn update(home_dir: &Path, request: &StoreUpdate) -> Outcome { + let name = &request.name; + let record = postgresql_record(home_dir, name)?; + record.unblocked()?; + if request.url.is_none() && request.password.is_none() && request.synchronous_commit.is_none() { + return Err(refused( + "InvalidInput", + "nothing to update: give --url, --password or --synchronous-commit", + )); + } + if let Some(url) = &request.url { + refuse_password(url)?; + } + if let Some(sync) = &request.synchronous_commit { + synchronous_commit(sync)?; + } + if let Some(password) = &request.password { + resolve(password)?; + } + let dialled = request.url.is_some() || request.password.is_some(); + if dialled { + let trial = changed(record.clone(), request); + let probe = selection(&trial)?; + if let Some(url) = &request.url { + unclaimed_by_others(home_dir, name, &target(url)?, record.binding())?; + } + same_store(&trial, &probe).map_err(|refusal| named(refusal, name))?; + } + let home_lock = home::lock::take(home_dir)?; + let store_lock = crate::record::lock(home_dir, name)?; + let current = StoreRecord::read(home_dir, name)?.filter(|r| r.binding() == record.binding()); + let Some(current) = current else { + return Err(refused( + "Conflict", + format!("Store {name} changed while its connection was checked; update it again"), + )); + }; + current.unblocked()?; + crate::open::close(¤t.postgresql_key(home_dir, name))?; + changed(current, request).write(home_dir)?; + drop((store_lock, home_lock)); + crate::sqlite::family::get(home_dir, name) +} + +/// `refusal` of the check, naming the Store. +fn named(refusal: SdkRefusal, name: &str) -> SdkRefusal { + refused( + &refusal.code, + format!("PostgreSQL Store {name}: {}", refusal.message), + ) +} diff --git a/control/crates/layerfs-store/src/record.rs b/control/crates/layerfs-store/src/record.rs index 91389cbe9b..ed948fa5f0 100644 --- a/control/crates/layerfs-store/src/record.rs +++ b/control/crates/layerfs-store/src/record.rs @@ -46,31 +46,34 @@ pub struct PgServer { pub synchronous_commit: Option, } -/// One machine a PostgreSQL Store is attached to, its record's +/// A machine's own connection to a PostgreSQL Store, its record's /// `machines.`: the URL (no password) and the password's reference -/// that machine dials when they are not the Store's, and its last check -/// that passed (`{ at, server_version, latency_ms }`). No `Debug`, as with -/// every value that names a password reference. +/// that machine dials where they are not the Store's (`store.update +/// --machine`). A machine without one dials the Store's. No `Debug`, as +/// with every value that names a password reference. #[derive(Clone, Default, PartialEq)] -pub struct Attachment { +pub struct OwnConnection { /// The URL the machine dials, when not the Store's. pub url: Option, /// The password's reference, when not the Store's. pub password: Option, - /// The last check that passed, as recorded. - pub checked: Option, } -impl Attachment { +impl OwnConnection { fn from_record(value: &SdkValue) -> Self { Self { url: value.get_text("url").map(str::to_owned), password: value.get_text("password").map(str::to_owned), - checked: value.get("checked").cloned(), } } - /// The attachment as its record holds it: only the fields it has. + /// Whether it names neither a URL nor a password: the Store's + /// connection, which no record entry is kept for. + pub fn is_empty(&self) -> bool { + self.url.is_none() && self.password.is_none() + } + + /// The connection as its record holds it: only the fields it has. pub fn to_record(&self) -> SdkValue { let mut fields = Vec::new(); if let Some(url) = &self.url { @@ -79,9 +82,6 @@ impl Attachment { if let Some(password) = &self.password { fields.push(("password".to_owned(), SdkValue::text(password))); } - if let Some(checked) = &self.checked { - fields.push(("checked".to_owned(), checked.clone())); - } SdkValue::Map(fields) } } @@ -339,48 +339,50 @@ impl StoreRecord { }) } - /// The machines a PostgreSQL Store is attached to, in record order. A - /// record without `machines` (written before attachments) is attached - /// to `native` alone, with no check recorded. - pub fn attachments(&self) -> Vec<(String, Attachment)> { + /// The machines with their own connection to a PostgreSQL Store, in + /// record order. An entry that names neither a URL nor a password (a + /// record written while machines were attached one by one) is none. + pub fn own_connections(&self) -> Vec<(String, OwnConnection)> { match self.fields.get("machines") { Some(SdkValue::Map(machines)) => machines .iter() - .map(|(machine, value)| (machine.clone(), Attachment::from_record(value))) + .map(|(machine, value)| (machine.clone(), OwnConnection::from_record(value))) + .filter(|(_, own)| !own.is_empty()) .collect(), - _ => vec![("native".to_owned(), Attachment::default())], + _ => Vec::new(), } } - /// The Store's attachment to `machine`, if it is attached there. - pub fn attachment(&self, machine: &str) -> Option { - let mut found = self.attachments().into_iter(); - found.find(|(m, _)| m == machine).map(|(_, a)| a) + /// `machine`'s own connection, if it has one. + pub fn own_connection(&self, machine: &str) -> Option { + let mut found = self.own_connections().into_iter(); + found.find(|(m, _)| m == machine).map(|(_, own)| own) } - /// The record with its attachment to `machine` set to `attachment`. - pub fn with_attachment(self, machine: &str, attachment: &Attachment) -> Self { - let mut machines: Vec<(String, SdkValue)> = self - .attachments() - .into_iter() - .map(|(m, a)| (m, a.to_record())) - .collect(); - match machines.iter_mut().find(|(m, _)| m == machine) { - Some(slot) => slot.1 = attachment.to_record(), - None => machines.push((machine.to_owned(), attachment.to_record())), + /// The record with `machine`'s own connection set to `own`; an empty + /// one removes it. + pub fn with_own_connection(self, machine: &str, own: &OwnConnection) -> Self { + let mut machines = self.own_connections(); + machines.retain(|(m, _)| m != machine); + if !own.is_empty() { + machines.push((machine.to_owned(), own.clone())); } - self.with("machines", SdkValue::Map(machines)) + self.with_machines(machines) } - /// The record without an attachment to `machine`. - pub fn without_attachment(self, machine: &str) -> Self { - let machines = self - .attachments() - .into_iter() - .filter(|(m, _)| m != machine) - .map(|(m, a)| (m, a.to_record())) - .collect(); - self.with("machines", SdkValue::Map(machines)) + /// The record without an own connection for `machine`. + pub fn without_own_connection(self, machine: &str) -> Self { + self.with_own_connection(machine, &OwnConnection::default()) + } + + /// The record with `machines` holding exactly these own connections, + /// and no `machines` field when there are none. + fn with_machines(self, machines: Vec<(String, OwnConnection)>) -> Self { + if machines.is_empty() { + return self.without("machines"); + } + let machines = machines.into_iter().map(|(m, own)| (m, own.to_record())); + self.with("machines", SdkValue::Map(machines.collect())) } fn raw_links(&self) -> &[(String, SdkValue)] { @@ -463,12 +465,12 @@ impl StoreRecord { } out.retain(|(field, _)| field != "name"); out.insert(0, ("name".to_owned(), SdkValue::text(&self.name))); - // a PostgreSQL record written before attachments shows the one it has - let postgresql = matches!(self.parsed, Ok(StoreRef::Postgres { .. })); - if postgresql && !out.iter().any(|(field, _)| field == "machines") { - let machines = self.attachments().into_iter(); - let machines = machines.map(|(m, a)| (m, a.to_record())).collect(); - out.push(("machines".to_owned(), SdkValue::Map(machines))); + // `machines` shows only the machines with their own connection + out.retain(|(field, _)| field != "machines"); + let own = self.own_connections(); + if !own.is_empty() { + let machines = own.into_iter().map(|(m, own)| (m, own.to_record())); + out.push(("machines".to_owned(), SdkValue::Map(machines.collect()))); } let names = derived.projects.iter().map(SdkValue::text).collect(); out.push(("projects".to_owned(), SdkValue::List(names))); @@ -524,10 +526,6 @@ impl StoreRecord { ("schema", SdkValue::text(schema)), ("synchronous_commit", SdkValue::text(synchronous_commit)), ("created", SdkValue::Int(home::now_ms())), - ( - "machines", - SdkValue::map([("native", SdkValue::Map(Vec::new()))]), - ), ]); let record = Self::from_record(name, fields); match password { @@ -561,11 +559,11 @@ pub fn url(record: &StoreRecord) -> Result { } /// The full URL machine `machine` dials for PostgreSQL Store `record`, its -/// password resolved now: its attachment's URL and password reference -/// where it has its own, else the Store's. +/// password resolved now: its own connection's URL and password reference +/// where it has one, else the Store's. pub fn url_on(record: &StoreRecord, machine: &str) -> Result { - let attachment = record.attachment(machine).unwrap_or_default(); - let url = match attachment.url.as_deref().or(record.text("url")) { + let own = record.own_connection(machine).unwrap_or_default(); + let url = match own.url.as_deref().or(record.text("url")) { Some(url) => url, None => { return Err(refused( @@ -574,7 +572,7 @@ pub fn url_on(record: &StoreRecord, machine: &str) -> Result { )) } }; - let password = attachment.password.as_deref().or(record.text("password")); + let password = own.password.as_deref().or(record.text("password")); layerfs_home::secrets::with_password(url, password) } diff --git a/docs/architecture/api/README.md b/docs/architecture/api/README.md index 6ba0a7f4bd..1429dc920a 100644 --- a/docs/architecture/api/README.md +++ b/docs/architecture/api/README.md @@ -2,7 +2,7 @@ > **Status:** Current architecture guide for LayerFS 2.0.0; maintained with the source. -The public surface: the `layerfs-api` call table with its one dispatcher, and the front ends (the `layerfs` command, Node, Python and a local MCP server) that expose its **85 calls in 12 families** with the same names and arguments (the MCP server lists 78: six monitoring calls are for a UI only, and `doctor.fix` is no tool). +The public surface: the `layerfs-api` call table with its one dispatcher, and the front ends (the `layerfs` command, Node, Python and a local MCP server) that expose its **84 calls in 12 families** with the same names and arguments (the MCP server lists 78: six monitoring calls are for a UI only, and `doctor.fix` is no tool). | | | | --- | --- | @@ -85,7 +85,7 @@ give each call's own arguments, answer and errors. | Test or example | Command | Platform | Shows | | --- | --- | --- | --- | -| [`calls_table.rs`](../../../api/crates/layerfs-api/tests/calls_table.rs) | `cargo +1.85.1 test --locked -p layerfs-api --test calls_table` | any Unix host | 85 calls in twelve families; one source file per call; refusals before any effect | +| [`calls_table.rs`](../../../api/crates/layerfs-api/tests/calls_table.rs) | `cargo +1.85.1 test --locked -p layerfs-api --test calls_table` | any Unix host | 84 calls in twelve families; one source file per call; refusals before any effect | | [`coverage.mjs`](../../../api/node/test/coverage.mjs) | `node --test api/node/test/coverage.mjs` | macOS or Linux, with the built addon, Python module and CLI | every front end matches the table | | [`file-history.mjs`](../../../api/node/examples/file-history.mjs) | `node api/node/examples/file-history.mjs` | macOS or Linux, with the built addon | a Store, Init, file writes and history reads | diff --git a/docs/architecture/api/calls/README.md b/docs/architecture/api/calls/README.md index c0efba82ae..5c999c6e82 100644 --- a/docs/architecture/api/calls/README.md +++ b/docs/architecture/api/calls/README.md @@ -43,7 +43,7 @@ The call table and the one dispatcher every front end goes through: what a call | Behaviour | Detail | Source | | --- | --- | --- | -| Table | twelve `Family { name, list }`; a `Call` is `{ family, call, args, options, help, run, quiet, opens }`, an `Arg` `{ name, kind, optional }`; 85 calls, help order | [`table.rs`](../../../../api/crates/layerfs-api/src/calls/table.rs), [`lib.rs`](../../../../api/crates/layerfs-api/src/lib.rs) `families` | +| Table | twelve `Family { name, list }`; a `Call` is `{ family, call, args, options, help, run, quiet, opens }`, an `Arg` `{ name, kind, optional }`; 84 calls, help order | [`table.rs`](../../../../api/crates/layerfs-api/src/calls/table.rs), [`lib.rs`](../../../../api/crates/layerfs-api/src/lib.rs) `families` | | JSON form | `table_json()`: `[{ family, call, args: [{ name, kind, optional }], options, help }]`; Node's `table`, Python's `TABLE`; the CLI parser and help read the same `Call` values | `table.rs` `table_json` | | Shared options | `PAGE` (`page_size`, `page_pos`), `DIFF` (`paths_only`, `page_size`, `page_pos`, `cursor`), `MESSAGE` (`message`), `PROGRESS` (`on_progress`), `WINDOW` (`cursor`, `before`, `limit`), `FILES` (`depth`, `cursor`, `limit`, `offset`, `length`) | `table.rs` | | Kinds | below | `table.rs` `Kind` | @@ -163,7 +163,7 @@ changes it. Reading and clearing it are the [`logs`](../sdk/logs.md) calls. | Test or example | Command | Platform | Shows | | --- | --- | --- | --- | -| [`calls_table.rs`](../../../../api/crates/layerfs-api/tests/calls_table.rs) | `cargo +1.85.1 test --locked -p layerfs-api --test calls_table` | any Unix host, temporary folders | twelve families and 85 calls; unknown calls and arguments refused before any effect; each family reached; JSON round trips; sizes and keys; a relative home refused; one source file per call | +| [`calls_table.rs`](../../../../api/crates/layerfs-api/tests/calls_table.rs) | `cargo +1.85.1 test --locked -p layerfs-api --test calls_table` | any Unix host, temporary folders | twelve families and 84 calls; unknown calls and arguments refused before any effect; each family reached; JSON round trips; sizes and keys; a relative home refused; one source file per call | | [`calls_scope.rs`](../../../../api/crates/layerfs-api/tests/calls_scope.rs) | `cargo +1.85.1 test --locked -p layerfs-api --test calls_scope` | any Unix host | a scope names its project, refuses every call it does not cover, needs a Sandbox of its project, and a scoped `branch.list` reads every page | | [`scope_references.rs`](../../../../api/crates/layerfs-api/tests/scope_references.rs) | `cargo +1.85.1 test --locked -p layerfs-api --test scope_references` | any Unix host | fork and publish references outside the namespace refused before any effect; the JSON form refuses the same | | [`logs.rs`](../../../../api/crates/layerfs-api/tests/logs.rs) | `cargo +1.85.1 test --locked -p layerfs-api --test logs` | any Unix host | one line per call, rendering and redaction, `logs.list` and `logs.reset` | diff --git a/docs/architecture/api/calls/coverage.md b/docs/architecture/api/calls/coverage.md index 84445a48cf..d7a178fa61 100644 --- a/docs/architecture/api/calls/coverage.md +++ b/docs/architecture/api/calls/coverage.md @@ -42,7 +42,7 @@ check and proves a missing, renamed or extra call is reported. -85 calls in 12 families, counted from the Node table. Calls present: Node 85, Python 85, Python `aio` 85, CLI 85, MCP 78 (38 in `work`, 40 admin only; 6 UI only, in no profile). +84 calls in 12 families, counted from the Node table. Calls present: Node 84, Python 84, Python `aio` 84, CLI 84, MCP 77 (38 in `work`, 39 admin only; 6 UI only, in no profile). Argument names that are Python keywords: none. @@ -114,12 +114,11 @@ Front-end problems: none. | `machine.update` | name | cpus, memory, image, docker, wsl, daemon | yes | yes | yes | yes | admin | | `store.list` | - | - | yes | yes | yes | yes | admin | | `store.get` | name | - | yes | yes | yes | yes | admin | -| `store.status` | name | - | yes | yes | yes | yes | admin | +| `store.status` | name | machine | yes | yes | yes | yes | admin | | `store.stats` | name | - | yes | yes | yes | yes | admin | | `store.create` | name | - | yes | yes | yes | yes | admin | | `store.connect` | name, url | password, synchronous_commit | yes | yes | yes | yes | admin | -| `store.attach` | name, machine | url, password | yes | yes | yes | yes | admin | -| `store.detach` | name, machine | - | yes | yes | yes | yes | admin | +| `store.update` | name | machine, url, password, synchronous_commit, clear | yes | yes | yes | yes | admin | | `store.push` | name, machine | on_progress() | yes | yes | yes | yes (--progress) | admin | | `store.pull` | name | on_progress() | yes | yes | yes | yes (--progress) | admin | | `store.remove` | name | - | yes | yes | yes | yes | admin | diff --git a/docs/architecture/api/front-ends/mcp.md b/docs/architecture/api/front-ends/mcp.md index 7fc9f79a20..d2c149c310 100644 --- a/docs/architecture/api/front-ends/mcp.md +++ b/docs/architecture/api/front-ends/mcp.md @@ -115,7 +115,7 @@ built addon). -85 calls: 38 in `work`, 40 in `admin` only, 6 in no profile (`ui`), 1 no tool; `--profile admin` lists 78. +84 calls: 38 in `work`, 39 in `admin` only, 6 in no profile (`ui`), 1 no tool; `--profile admin` lists 77. | Call | Bucket | Profiles that list it | Why | | --- | --- | --- | --- | @@ -187,8 +187,7 @@ built addon). | `store.stats` | admin | admin | create, connect, transfer and remove Stores; administration | | `store.create` | admin | admin | create, connect, transfer and remove Stores; administration | | `store.connect` | admin | admin | create, connect, transfer and remove Stores; administration | -| `store.attach` | admin | admin | create, connect, transfer and remove Stores; administration | -| `store.detach` | admin | admin | create, connect, transfer and remove Stores; administration | +| `store.update` | admin | admin | create, connect, transfer and remove Stores; administration | | `store.push` | admin | admin | create, connect, transfer and remove Stores; administration | | `store.pull` | admin | admin | create, connect, transfer and remove Stores; administration | | `store.remove` | admin | admin | create, connect, transfer and remove Stores; administration | diff --git a/docs/architecture/api/sdk/README.md b/docs/architecture/api/sdk/README.md index 8d21e23e92..55ea37face 100644 --- a/docs/architecture/api/sdk/README.md +++ b/docs/architecture/api/sdk/README.md @@ -2,7 +2,7 @@ > **Status:** Current architecture guide for LayerFS 2.0.0; maintained with the source. -Every call a program can make: **85 calls in 12 families**, the same names and arguments in JavaScript, Python and the `layerfs` command, one page per family. +Every call a program can make: **84 calls in 12 families**, the same names and arguments in JavaScript, Python and the `layerfs` command, one page per family. | | | | --- | --- | @@ -58,7 +58,7 @@ built table. | [`sandbox`](sandbox.md) | [`create`](sandbox.md#sandboxcreate), [`list`](sandbox.md#sandboxlist), [`get`](sandbox.md#sandboxget), [`close`](sandbox.md#sandboxclose), [`files`](sandbox.md#sandboxfiles) | records here; start and stop on the Sandbox's machine; `files`: where the Sandbox's folder is | [`sandbox/`](../../../../api/crates/layerfs-api/src/sandbox) | | [`workspace`](workspace.md) | [`mount`](workspace.md#workspacemount), [`exec`](workspace.md#workspaceexec), [`exec_cancel`](workspace.md#workspaceexec_cancel), [`commit`](workspace.md#workspacecommit), [`unmount`](workspace.md#workspaceunmount), [`status`](workspace.md#workspacestatus) | the Sandbox's daemon; `exec` and `exec_cancel` in the Sandbox's runtime | [`workspace/`](../../../../api/crates/layerfs-api/src/workspace) | | [`machine`](machine.md) | [`list`](machine.md#machinelist), [`platforms`](machine.md#machineplatforms), [`get`](machine.md#machineget), [`images`](machine.md#machineimages), [`load`](machine.md#machineload), [`prune`](machine.md#machineprune), [`status`](machine.md#machinestatus), [`stats`](machine.md#machinestats), [`space`](machine.md#machinespace), [`files`](machine.md#machinefiles), [`start`](machine.md#machinestart), [`stop`](machine.md#machinestop), [`create`](machine.md#machinecreate), [`remove`](machine.md#machineremove), [`update`](machine.md#machineupdate) | the caller's process, through the platform adapter; `stats`, `space`, `files`: where the machine's data is | [`machine/`](../../../../api/crates/layerfs-api/src/machine) | -| [`store`](store.md) | [`list`](store.md#storelist), [`get`](store.md#storeget), [`status`](store.md#storestatus), [`stats`](store.md#storestats), [`create`](store.md#storecreate), [`connect`](store.md#storeconnect), [`attach`](store.md#storeattach), [`detach`](store.md#storedetach), [`push`](store.md#storepush), [`pull`](store.md#storepull), [`remove`](store.md#storeremove), [`compact`](store.md#storecompact), [`disconnect`](store.md#storedisconnect) | the caller's process; sqlite file steps in the Store machine's worker | [`store/`](../../../../api/crates/layerfs-api/src/store) | +| [`store`](store.md) | [`list`](store.md#storelist), [`get`](store.md#storeget), [`status`](store.md#storestatus), [`stats`](store.md#storestats), [`create`](store.md#storecreate), [`connect`](store.md#storeconnect), [`update`](store.md#storeupdate), [`push`](store.md#storepush), [`pull`](store.md#storepull), [`remove`](store.md#storeremove), [`compact`](store.md#storecompact), [`disconnect`](store.md#storedisconnect) | the caller's process; sqlite file steps in the Store machine's worker | [`store/`](../../../../api/crates/layerfs-api/src/store) | | [`backup`](backup.md) | [`connect`](backup.md#backupconnect), [`disconnect`](backup.md#backupdisconnect), [`list`](backup.md#backuplist), [`create`](backup.md#backupcreate), [`restore`](backup.md#backuprestore) | the caller's process; `create` stages on the Store's machine | [`backup/`](../../../../api/crates/layerfs-api/src/backup) | | [`logs`](logs.md) | [`list`](logs.md#logslist), [`reset`](logs.md#logsreset), [`machine`](logs.md#logsmachine), [`sandbox`](logs.md#logssandbox) | `list`, `reset`: the caller's process, home files only; `machine`, `sandbox`: where the log file is | [`logs/`](../../../../api/crates/layerfs-api/src/logs) | | [`doctor`](doctor.md) | [`fix`](doctor.md#doctorfix) | the machine its id names: the caller's process on `native`, else that machine's worker | [`doctor/`](../../../../api/crates/layerfs-api/src/doctor) | @@ -106,7 +106,7 @@ in the [API layer page](../README.md#calling-conventions). | Test or example | Command | Platform | Shows | | --- | --- | --- | --- | | [`inventory.mjs`](../../../../api/node/test/branch-file/inventory.mjs) | `perl -e 'alarm 120; exec @ARGV' node --test api/node/test/branch-file/inventory.mjs` | macOS or Linux, with the built addon (`npm --prefix api run build:native`) and `target/debug/layerfs` | the count line and one heading per call match the built table; branch and file CLI help exposes every option | -| [`calls_table.rs`](../../../../api/crates/layerfs-api/tests/calls_table.rs) | `cargo +1.85.1 test --locked -p layerfs-api --test calls_table` | any Unix host | 85 calls in twelve families; one source file per call; unknown calls and arguments refused before any effect | +| [`calls_table.rs`](../../../../api/crates/layerfs-api/tests/calls_table.rs) | `cargo +1.85.1 test --locked -p layerfs-api --test calls_table` | any Unix host | 84 calls in twelve families; one source file per call; unknown calls and arguments refused before any effect | | [`file-history.mjs`](../../../../api/node/examples/file-history.mjs) | `node api/node/examples/file-history.mjs` | macOS or Linux, with the built addon | a Store, Init, file writes and history reads | ## Pages diff --git a/docs/architecture/api/sdk/machine.md b/docs/architecture/api/sdk/machine.md index 102f1b1fc0..da4d85ad88 100644 --- a/docs/architecture/api/sdk/machine.md +++ b/docs/architecture/api/sdk/machine.md @@ -50,7 +50,7 @@ The **summary** is `MachineRecord::to_sdk` ```text { name, platform, backend, supports: { sqlite, postgresql_sandbox, reason, log }, - state, cpus, memory, disk, stores: [name], postgresql: [name], + state, cpus, memory, disk, stores: [name], sandboxes: [name], image?, docker?, wsl?, daemon?, os_image?, error?, ssh?, host?, vm_name? } ``` @@ -62,7 +62,7 @@ The **summary** is `MachineRecord::to_sdk` | `supports` | the platform's support row on this host; a removed platform supports nothing, with the removal as `reason` | | `supports.log` | whether the machine has a worker log to read (layerfs-project#178): its calls go to a worker and the support table refuses it nothing on this host, so it is a worker this build starts. True for `docker`, `lima` and `apple-container` where they run; false on every `native` (its calls run in the caller's process; on Windows its WSL2 worker is not designed) and for a removed platform. Clients read this field and never test platform names (`Supports::log`, [`support.rs`](../../../../control/crates/layerfs-platform/src/support.rs)) | | `cpus`, `memory`, `disk` | the recorded size (bytes for memory and disk); null on a built-in machine | -| `stores`, `postgresql`, `sandboxes` | sqlite Stores on it, PostgreSQL Stores attached to it, Sandboxes recorded on it ([`inventory.rs`](../../../../control/crates/layerfs-control/src/inventory.rs)) | +| `stores`, `sandboxes` | sqlite Stores on it, Sandboxes recorded on it (a PostgreSQL Store is on no machine) ([`inventory.rs`](../../../../control/crates/layerfs-control/src/inventory.rs)) | | `error` | the probe's reason when `unknown`; the removal for a removed platform; a failed create's error | | `state` | When | @@ -641,7 +641,7 @@ in its Lima host) ([`remove.rs`](../../../../control/crates/layerfs-machine/src/ | Error | When | | --- | --- | | `Busy` | a live process's create or remove holds it (a gone process's uncertain intent does not refuse) | -| `InUse` | sqlite Stores, attached PostgreSQL Stores, Sandboxes or old Firecracker records it hosted are on it (`detail.related`, `can_remove_machine`) | +| `InUse` | sqlite Stores, Sandboxes or old Firecracker records it hosted are on it (`detail.related`, `can_remove_machine`) | | `Unavailable`, `Failed` | the provider refused or failed; the record is put back | | `Unknown` | the provider timed out: `op` kept, uncertain; read `machine.status`, then remove again | diff --git a/docs/architecture/api/sdk/store.md b/docs/architecture/api/sdk/store.md index c1a222c86d..df826db40d 100644 --- a/docs/architecture/api/sdk/store.md +++ b/docs/architecture/api/sdk/store.md @@ -2,13 +2,13 @@ > **Status:** Current architecture guide for LayerFS 2.0.0; maintained with the source. -The `store` calls: create, connect, inspect, move, remove and compact the databases that hold every project's history, within the [SDK](README.md). +The `store` calls: create, connect, update, inspect, move, remove and compact the databases that hold every project's history, within the [SDK](README.md). | | | | --- | --- | | Source | [`api/crates/layerfs-api/src/store`](../../../../api/crates/layerfs-api/src/store), [`layerfs-store/src`](../../../../control/crates/layerfs-store/src) (`sqlite/`, `postgresql/`, `list.rs`, `status.rs`, `stats.rs`), [`layerfs-control/src/store.rs`](../../../../control/crates/layerfs-control/src/store.rs) | | Runs by default | the caller's process (records, locks, leases, PostgreSQL connections); a sqlite Store's file steps run in its machine's worker (`store.file`) when it is not on a locally executing machine | -| Design | [Stores and projects](../../control/stores-and-projects/README.md) (records, placement, leases), [SQLite and PostgreSQL](../../control/stores-and-projects/sqlite-and-postgresql.md) (transfer phases, connect and attach), [Engine: Store](../../engine/store/README.md) | +| Design | [Stores and projects](../../control/stores-and-projects/README.md) (records, placement, leases), [SQLite and PostgreSQL](../../control/stores-and-projects/sqlite-and-postgresql.md) (transfer phases, connect, update and the check), [Engine: Store](../../engine/store/README.md) | ## Calls @@ -18,11 +18,11 @@ The `store` calls: create, connect, inspect, move, remove and compact the databa sqlite | PostgreSQL ------ | ---------- create (on native) | connect (url, password ref) - | | | attached to native + | | | runs on every machine v | v - push --> on --> pull attach [url, password] - | (back to native) detach - v v + push --> on --> pull update [url, password, sync] + | (back to native) update --machine (its own + v v connection) remove (file + projects) disconnect (data stays) compact (reclaim dead bytes) ``` @@ -31,12 +31,11 @@ The `store` calls: create, connect, inspect, move, remove and compact the databa | --- | --- | --- | | [`store.list`](#storelist) | both | `{ items: [record + projects + state] }` | | [`store.get`](#storeget) | both | the record with `projects` | -| [`store.status`](#storestatus) | both | `{ name, kind, state, holders, op, down_reason, machine/file \| server }` | +| [`store.status`](#storestatus) | both | `{ name, kind, state, holders, op, down_reason, machine/file \| server, check? }` | | [`store.stats`](#storestats) | both | publication and allocation counters | | [`store.create`](#storecreate) | sqlite | the new record | -| [`store.connect`](#storeconnect) | PostgreSQL | `{ name, kind, schema, synchronous_commit, machines, projects }` | -| [`store.attach`](#storeattach) | PostgreSQL | the Store as `get` shows it | -| [`store.detach`](#storedetach) | PostgreSQL | the Store as `get` shows it | +| [`store.connect`](#storeconnect) | PostgreSQL | `{ name, kind, schema, synchronous_commit, projects }` | +| [`store.update`](#storeupdate) | PostgreSQL | the Store as `get` shows it | | [`store.push`](#storepush) | sqlite | `{ store, from, to, bytes, heads }` | | [`store.pull`](#storepull) | sqlite | `{ store, from, to, bytes, heads }` | | [`store.remove`](#storeremove) | sqlite | `{ removed: name }` | @@ -62,7 +61,8 @@ shows `cursor`, `binding` or `v`: | `last_transfer` | sqlite | the last push or pull `{ call, from, to, at, bytes, sha256, heads, retired }`; `retired: false` means the source copy is not yet deleted | | `incoming_on` | sqlite | a machine a certain-failed push left a partial `.incoming` copy on; the next push or remove deletes it ([`sqlite/transfer/resolve.rs`](../../../../control/crates/layerfs-store/src/sqlite/transfer/resolve.rs)) | | `backups` | sqlite | one link per bucket: the file, its heads, the last `create` or `restore` ([`backup/record.rs`](../../../../control/crates/layerfs-store/src/backup/record.rs)) | -| `url`, `schema`, `synchronous_commit`, `password`, `machines` | PostgreSQL | the URL without a password, always `layerfs`, the setting, the password reference, attached machines `{ url?, password?, checked? }` | +| `url`, `schema`, `synchronous_commit`, `password` | PostgreSQL | the URL without a password, always `layerfs`, the setting, the password reference | +| `machines` | PostgreSQL | only the machines with a connection of their own, `{ : { url?, password? } }`; absent when none has one. Every other machine dials the Store's | | `op` | both | the recorded intent `{ id, name, phase, started, uncertain }`; for a compaction `id` is the 16-byte operation id rendered `op-<32 lowercase hex>` ([`op.rs`](../../../../control/crates/layerfs-home/src/op.rs), [`status.rs`](../../../../control/crates/layerfs-store/src/status.rs)) | | `projects` | both | the home's project records for this Store, sorted | @@ -106,12 +106,13 @@ Errors: `NotFound`, `Failed` (the record cannot be read). ### store.status One Store's live state: who holds it, the operation on it, and its file or -server. Changes nothing. +server. With `machine`, also whether that machine reaches a PostgreSQL +Store. Changes nothing. ```text -JS await layerfs.store.status(name) -Python layerfs.store.status(name) -CLI layerfs store status +JS await layerfs.store.status(name, { machine }) +Python layerfs.store.status(name, *, machine=None) +CLI layerfs store status [--machine ] ``` ```text @@ -122,15 +123,32 @@ CLI layerfs store status latency_us, schemas: { layerfs: | null } } | { reachable: false, error: { code, message } } | { unavailable } + --machine: check: { machine, host, ok: true, at, server_version, latency_ms } + | { machine, host, ok: false, error: { code, message } } ``` +`check` is the check of one machine's connection (its own where it has +one, else the Store's), run from that machine now: in this process, or in +its worker as `store.reach`. It reads the server version, the role, the +`layerfs` schema and its binding, `USAGE` on the schema, `TEMP` on the +database, `SELECT`/`INSERT`/`UPDATE`/`DELETE` on each table and `USAGE` on +each sequence ([`access.rs`](../../../../engine/crates/layerfs-persistence/src/backend/postgresql/access.rs), +[`postgresql/check.rs`](../../../../control/crates/layerfs-store/src/postgresql/check.rs)). +A check that fails is `ok: false` in a successful answer, with the refusal +`store.update --machine` would give (`Unavailable`: not reached, or the +database holds another Store; `Denied`: the login or a privilege; a stopped +machine). Nothing is recorded. `host` is null when the URL the machine +dials could not be built ([`store.rs`](../../../../control/crates/layerfs-control/src/store.rs) `status`). + `holders` are live processes other than this one holding its lease; stale leases are ignored, not removed ([`lease.rs`](../../../../control/crates/layerfs-store/src/lease.rs)). The password reference is resolved here and the server dialed once; a PostgreSQL Store is never `down` (an unreachable server shows in `server`) ([`status.rs`](../../../../control/crates/layerfs-store/src/status.rs)). It is -the call an `Unresolved` refusal names. When a compaction runs on a sqlite +the call an `Unresolved` refusal names. With `machine`: `InvalidInput` (a +sqlite Store), `NotFound` (no such machine), `Unsupported` (its platform +runs no PostgreSQL Sandboxes here). When a compaction runs on a sqlite Store, `op` is read from `.compact-op`; on a PostgreSQL Store it is the Store-side `gc_op` row with its `gc_intent`, read by one status query with no Store session and no artifact read. Either way `state` is `busy` only @@ -200,9 +218,10 @@ user, is a symlink, or has group/other access instead of `0700`); ### store.connect -Attaches a PostgreSQL database as a Store: an empty database gets the +Connects a PostgreSQL database as a Store: an empty database gets the `layerfs` schema; an existing one keeps its data and its projects become this -home's project records. +home's project records. Its Sandboxes then run on every machine whose +platform runs PostgreSQL Sandboxes, with no further step. ```text JS await layerfs.store.connect(name, url, { password, synchronous_commit }) @@ -218,13 +237,14 @@ CLI layerfs store connect [--password ] [--synchronou ```text { name, kind: "postgresql", schema: "layerfs", synchronous_commit, - machines: { native: { checked? } }, projects: [name] } + projects: [name] } ``` -`native` is attached with the Store's own connection; `checked` is present -when the `store.attach` check passed. The schema is made before any record is -written, so a refusal after it leaves an empty schema and no record; connecting -again keeps its binding +No machine is recorded: each machine dials the Store's connection (a +loopback host rewritten for it, [Reach](../../control/machines/workers-and-reach.md#reach-reachrs)), +and `sandbox.create` checks from the machine that it reaches this Store. +The schema is made before any record is written, so a refusal after it +leaves an empty schema and no record; connecting again keeps its binding ([`postgresql/connect.rs`](../../../../control/crates/layerfs-store/src/postgresql/connect.rs)). | Error | When | @@ -233,55 +253,44 @@ again keeps its binding | `Unavailable` | the reference does not resolve; no session (no database, wrong user or password); a lock held 120 s | | `Exists` | the name is taken, or another PostgreSQL Store of this home connects the same database (by URL, or by binding once read) | -### store.attach +### store.update -Lets a machine run Sandboxes of a PostgreSQL Store, after checking from that -machine that it reaches the database and its role may write there. +Changes a PostgreSQL Store's connection, or gives one machine a connection +of its own. ```text -JS await layerfs.store.attach(name, machine, { url, password }) -Python layerfs.store.attach(name, machine, *, url=None, password=None) -CLI layerfs store attach [--url ] [--password ] +JS await layerfs.store.update(name, { machine, url, password, synchronous_commit, clear }) +Python layerfs.store.update(name, *, machine=None, url=None, password=None, synchronous_commit=None, clear=None) +CLI layerfs store update [--machine ] [--url ] [--password ] [--synchronous-commit ] [--clear] ``` -`machine` must be a platform that runs PostgreSQL Sandboxes on this host (the -support table, as `sandbox.create`). `url` and `password` replace the Store's -for that machine, under `store.connect`'s rules; either may be given alone. -On an attached machine it checks the new connection and replaces the old one; -with no options it checks the current one again. The check runs where the -machine's Sandbox calls run (here, or the worker as `store.reach`): server -version, role, the `layerfs` schema, `USAGE` on it, `TEMP` on the database, -`SELECT`/`INSERT`/`UPDATE`/`DELETE` on each table and `USAGE` on each -sequence ([`access.rs`](../../../../engine/crates/layerfs-persistence/src/backend/postgresql/access.rs)). -A failed check writes nothing; LayerFS never creates or grants roles -([`store.rs`](../../../../control/crates/layerfs-control/src/store.rs), -[`postgresql/attach.rs`](../../../../control/crates/layerfs-store/src/postgresql/attach.rs)). +| Form | Does | +| --- | --- | +| no `machine` | sets the Store's `url`, `password` reference and `synchronous_commit` (any of them). A new URL or password is dialled once from this process and must reach this Store: the same binding. `synchronous_commit` alone dials nothing | +| `machine` with `url` and/or `password` | sets the connection that machine alone dials, in place of the Store's; a field not given keeps the machine's own, else the Store's. Checked from the machine first, as [`store.status --machine`](#storestatus) checks it | +| `machine` with `clear` | removes the machine's own connection; it dials the Store's again. Dials nothing | + +`url` and `password` follow `store.connect`'s rules. Every refusal of the +input comes before any connection, and a failed check saves nothing. The +record is written under the home lock and the Store's; this process's open +sessions on the Store are closed. A running Sandbox keeps the connection it +started with. LayerFS never creates or grants roles. Removing a machine +forgets its own connection in every Store +([`postgresql/update.rs`](../../../../control/crates/layerfs-store/src/postgresql/update.rs), +[`postgresql/check.rs`](../../../../control/crates/layerfs-store/src/postgresql/check.rs), +[`store.rs`](../../../../control/crates/layerfs-control/src/store.rs)). | Error | When | | --- | --- | -| `InvalidInput` | a sqlite Store (names `store push`); a URL holding a password | -| `Unsupported` | the platform runs no PostgreSQL Sandboxes here; an ssh loopback URL; a worker bundle without `store.reach` | -| `Unavailable` | the reference does not resolve; the machine is stopped; the server is not reached from it, or its database holds another Store or none (names the machine and host, never the URL) | -| `Denied` | the server refused the login, or the role lacks a privilege (names the first and how many) | +| `InvalidInput` | a sqlite Store; nothing to update; a URL holding a password; a bad `synchronous_commit`; `synchronous_commit` with `machine`; `clear` without `machine`, with `url` or `password`, or for a machine with no connection of its own | +| `NotFound` | no such Store or machine | +| `Unsupported` | the machine's platform runs no PostgreSQL Sandboxes here; a URL the platform cannot rewrite for the machine; a worker bundle without `store.reach` | +| `Unavailable` | the reference does not resolve; the machine is stopped; the server is not reached, or its database holds another Store or none (names the machine and host, never the URL) | +| `Denied` | for a machine: the server refused the login, or the role lacks a privilege (names the first and how many) | +| `Exists` | the new URL names a database another Store of this home connects | +| `InUse` | a call of this process is running on the Store | | `Conflict` | the Store was disconnected or reconnected elsewhere during the check | -### store.detach - -Stops a machine running Sandboxes of a PostgreSQL Store. The database and -the Store's other machines are untouched. - -```text -JS await layerfs.store.detach(name, machine) -Python layerfs.store.detach(name, machine) -CLI layerfs store detach -``` - -Dials nothing. Detaching `native` is allowed: Store calls still run in the -caller with the Store's connection; only Sandboxes on `native` are refused -afterwards. Errors: `InvalidInput` (a sqlite Store, or not attached there); -`InUse` (a Sandbox of the Store is recorded on the machine, `detail.related`, -[`inventory.rs`](../../../../control/crates/layerfs-control/src/inventory.rs) `store_idle_on`). - ### store.push Moves a sqlite Store's file from `native` to another machine, checked byte diff --git a/docs/architecture/control/README.md b/docs/architecture/control/README.md index 303492b770..2eabebba22 100644 --- a/docs/architecture/control/README.md +++ b/docs/architecture/control/README.md @@ -74,9 +74,9 @@ machine's worker, or in a Sandbox's daemon. | `store.remove` | records, locks | `store.file` `remove` | — | [`sqlite/remove.rs`](../../../control/crates/layerfs-store/src/sqlite/remove.rs) | | `store.status`, `store.list` | records, leases | `store.file` `stat`, when reached | — | [`status.rs`](../../../control/crates/layerfs-store/src/status.rs), [`list.rs`](../../../control/crates/layerfs-store/src/list.rs) | | `store.compact` | all of it on the Store's own machine: PostgreSQL dials its URL, a sqlite file here rebuilds in this process | —: never forwarded (a remote sqlite Store is `Unsupported`, the documented S6 deviation #42 D5) | — | [`compact.rs`](../../../control/crates/layerfs-store/src/compact.rs) | -| `store.attach` | records; dials itself for `native` | `store.reach`: the machine's own check | — | [`layerfs-control/src/store.rs`](../../../control/crates/layerfs-control/src/store.rs) | +| `store.update`, `store.status --machine` | records; a Store's own connection and a `native` check are dialled here | `store.reach`: the machine's own check | — | [`layerfs-control/src/store.rs`](../../../control/crates/layerfs-control/src/store.rs) | | `backup.create` | uploads to the bucket | `store.file` `stage`, `read`, `unstage` | — | [`backup/create.rs`](../../../control/crates/layerfs-store/src/backup/create.rs) | -| `machine.*`, `store.create`, `store.connect`, `store.disconnect`, `store.detach`, `backup.connect`, `backup.disconnect`, `backup.list`, `backup.restore` | all of it | — | — | [Machines](machines/README.md), [Backups](backups/README.md) | +| `machine.*`, `store.create`, `store.connect`, `store.disconnect`, `backup.connect`, `backup.disconnect`, `backup.list`, `backup.restore` | all of it | — | — | [Machines](machines/README.md), [Backups](backups/README.md) | | `sandbox.create`, `sandbox.close` | records, intent | `sandbox.host` (the worker runs the same host code on its `native`) | started, stopped | [`route.rs`](../../../control/crates/layerfs-sandbox/src/route.rs) | | `workspace.mount`, `commit`, `unmount`, `status` on a forwarded Sandbox | dials the daemon directly | — | serves it | [`direct.rs`](../../../control/crates/layerfs-sandbox/src/direct.rs) | | the same, not forwarded | — (local machine: dials the daemon) | `sandbox.host`; the worker dials the daemon | serves it | `route.rs` | @@ -125,7 +125,7 @@ kind or kernel route is an engine change. | Rule | Enforced by | | --- | --- | -| sqlite Store calls run in a process on the Store's kernel (the caller on a local `native`, else that machine's worker); PostgreSQL Store calls and bucket I/O run in the caller's process; a sqlite Store is on exactly one machine; a PostgreSQL Store's Sandboxes run on the machines it is attached to | [`place.rs`](../../../control/crates/layerfs-store/src/place.rs), [`support.rs`](../../../control/crates/layerfs-platform/src/support.rs) | +| sqlite Store calls run in a process on the Store's kernel (the caller on a local `native`, else that machine's worker); PostgreSQL Store calls and bucket I/O run in the caller's process; a sqlite Store is on exactly one machine; a PostgreSQL Store is on no machine: its Sandboxes run on every machine whose platform runs them | [`place.rs`](../../../control/crates/layerfs-store/src/place.rs), [`support.rs`](../../../control/crates/layerfs-platform/src/support.rs) | | Workers listen on loopback behind a secret; one attempt per call, no retry, no replay; an uncertain outcome is reported, never resent | [`worker/client.rs`](../../../control/crates/layerfs-machine/src/worker/client.rs) | | A PostgreSQL password is resolved only in the caller, travels only in one worker request and the Sandbox's private daemon setup file, and is never written to a record, log, trace or `Debug` output | [`secrets.rs`](../../../control/crates/layerfs-home/src/secrets.rs), [SQLite and PostgreSQL](stores-and-projects/sqlite-and-postgresql.md#credentials) | | Drivers know nothing of the home and write no records | [`layerfs-platform/src/lib.rs`](../../../control/crates/layerfs-platform/src/lib.rs) | diff --git a/docs/architecture/control/home/migration.md b/docs/architecture/control/home/migration.md index 56ee1e10cc..5cec82db88 100644 --- a/docs/architecture/control/home/migration.md +++ b/docs/architecture/control/home/migration.md @@ -125,12 +125,11 @@ The relationships no record stores, read when a call needs them | Read or check | Answers | | --- | --- | -| `machines`, `stores_on`, `postgresql_on`, `sandboxes_on`, `hosted_on` | every machine record; the sqlite Stores, attached PostgreSQL Stores and Sandboxes of a machine; old Firecracker records a Lima machine hosted | +| `machines`, `stores_on`, `sandboxes_on`, `hosted_on` | every machine record; the sqlite Stores and Sandboxes of a machine; old Firecracker records a Lima machine hosted | | `sandboxes_of_store`, `sandboxes_of_project` | the Sandboxes of a Store or project | | `can_remove_machine` | `machine.remove`: `InUse` while it holds Stores, attached PostgreSQL Stores, Sandboxes or hosted Firecracker records | | `can_stop_machine`, `can_change_machine` | `machine.stop`, `machine.update`: `InUse` while Sandboxes are recorded on it | | `store_idle` | `store.push`, `pull`, `remove`, `disconnect`, `backup.restore`: `InUse` while another live process holds a lease or a project's Sandbox is recorded (removes stale leases) | -| `store_idle_on` | `store.detach`: `InUse` while a Sandbox of the Store is recorded on that machine | | `can_remove_project` | `project.delete`: `Busy` while a Sandbox of the project is recorded | Each `InUse` or `Busy` lists what holds the entity in `detail.related` as diff --git a/docs/architecture/control/machines/workers-and-reach.md b/docs/architecture/control/machines/workers-and-reach.md index b7a4c14c27..7d73638abe 100644 --- a/docs/architecture/control/machines/workers-and-reach.md +++ b/docs/architecture/control/machines/workers-and-reach.md @@ -161,7 +161,7 @@ waits at most 10 s; other operations wait as long as they run | `store.call` | one Store method on `store//store.sqlite` under the root, holding the Store's lease there | | `store.compact` | none: there is no compaction operation. A remote sqlite Store's `store.compact` is refused `Unsupported` before the worker is dialled (spec S6 deviation D5) | | `store.file` | one step of a Store push, pull, remove, stat or backup, with `progress` events for hashing steps | -| `store.reach` | `store.attach`'s check of this machine's connection to a PostgreSQL Store | +| `store.reach` | the check of this machine's connection to a PostgreSQL Store, for `store.update --machine` and `store.status --machine` | | `store.diagnose`, `doctor.sandbox` | doctor's read-only observations; `doctor.sandbox` with `leftovers: true` is the bounded walk for what interrupted operations left under the root | | `doctor.fix` | removes one such leftover, named by kind and name (never a path), under its guard ([Checks](../doctor/checks.md#leftovers)) | | `sandbox.host` | a Sandbox or Workspace host operation on the machine's own `native`, decoded into its typed request first | diff --git a/docs/architecture/control/stores-and-projects/sqlite-and-postgresql.md b/docs/architecture/control/stores-and-projects/sqlite-and-postgresql.md index 3cb4f22304..7098fd3435 100644 --- a/docs/architecture/control/stores-and-projects/sqlite-and-postgresql.md +++ b/docs/architecture/control/stores-and-projects/sqlite-and-postgresql.md @@ -3,8 +3,8 @@ > **Status:** Current architecture guide for LayerFS 2.0.0; maintained with the source. The two Store kinds' own calls: a sqlite Store's file created, moved and -removed across machines, and a PostgreSQL database connected, attached to -machines and opened, within [Stores and projects](README.md). +removed across machines, and a PostgreSQL database connected, updated, +checked from a machine and opened, within [Stores and projects](README.md). | | | | --- | --- | @@ -84,22 +84,29 @@ another operation holds the Store (once read) same schema binding schema `layerfs` empty database: create it, new binding; existing: read its binding - attach native checked with the Store's connection projects in the schema --> project records; record written under the home lock (re-checked) ``` | Call | Does | Refusals | Source | | --- | --- | --- | --- | -| `store.connect` | records the database as a Store in the fixed `layerfs` schema: URL without password, schema, `synchronous_commit`, password reference, binding, a cursor key of this home's own, `machines: {native: ..}` | `InvalidInput`, `Exists`, `Unavailable` | [`connect.rs`](../../../../control/crates/layerfs-store/src/postgresql/connect.rs) | +| `store.connect` | records the database as a Store in the fixed `layerfs` schema: URL without password, schema, `synchronous_commit`, password reference, binding, a cursor key of this home's own, no machine | `InvalidInput`, `Exists`, `Unavailable` | [`connect.rs`](../../../../control/crates/layerfs-store/src/postgresql/connect.rs) | | `store.disconnect` | forgets the Store; its data stays | `InvalidInput` for sqlite (`store remove`); `InUse` while leased or a Sandbox uses it | same | -| `store.attach {url, password}` | the machine's platform must run PostgreSQL Sandboxes (support table); the check runs on the side that dials (this process, or the worker's `store.reach`); records `machines.: {url?, password?, checked: {at, server_version, latency_ms}}` | `Unsupported` (row reason; a worker without `store.reach` names stop and start); `Unavailable` (server not reached, naming the host, never the URL, plus the platform's hint; or another Store's schema); `Denied` (login or privilege); `Conflict` if the record changed meanwhile | [`attach.rs`](../../../../control/crates/layerfs-store/src/postgresql/attach.rs), [`layerfs-control/src/store.rs`](../../../../control/crates/layerfs-control/src/store.rs) | -| `store.detach ` | forgets one attachment; data and other machines stay | `InvalidInput` if not attached; `InUse` while a Sandbox of the Store is recorded there | [`attach.rs`](../../../../control/crates/layerfs-store/src/postgresql/attach.rs) | +| `store.update {url, password, synchronous_commit}` | changes the Store's own connection; a new URL or password is dialled once from here and must hold this Store's binding; this process's open sessions on it are closed | `InvalidInput`; `Exists` (another Store's database); `Unavailable` (not reached, or another Store's schema); `InUse` (a call running here); `Conflict` if the record changed meanwhile | [`update.rs`](../../../../control/crates/layerfs-store/src/postgresql/update.rs) | +| `store.update --machine {url, password}` | gives one machine a connection of its own, `machines.: {url?, password?}`: the machine's platform must run PostgreSQL Sandboxes (support table); the check runs on the side that dials (this process, or the worker's `store.reach`) before anything is saved. `--clear` removes it and dials nothing | `Unsupported` (row reason; a worker without `store.reach` names stop and start); `Unavailable` (server not reached, naming the host, never the URL, plus the platform's hint; or another Store's schema); `Denied` (login or privilege); `Conflict` if the record changed meanwhile | [`check.rs`](../../../../control/crates/layerfs-store/src/postgresql/check.rs), [`layerfs-control/src/store.rs`](../../../../control/crates/layerfs-control/src/store.rs) | +| `store.status --machine ` | runs that check now and answers it as `check: {machine, host, ok, ..}`; records nothing | a failed check is `ok: false`, not a refusal | same | | Store calls | open the schema in this process with the URL resolved now | — | [`open.rs`](../../../../control/crates/layerfs-store/src/postgresql/open.rs) | | Host read session | one read-only session per process for Sandbox host operations: a create's reach check and a status's head read, on every route; opened with the light check on first read, checked with one empty query before each use (a dead one is replaced, not retried), closed after 60 s unused | — | [`reader.rs`](../../../../control/crates/layerfs-store/src/postgresql/reader.rs) | -A record without `machines` (written before attachments) is attached to -`native` alone. Nothing converts a sqlite Store into PostgreSQL: a project +A PostgreSQL Store is on no machine. Its Sandboxes run on every machine +whose platform runs PostgreSQL Sandboxes (#536 C4), and `sandbox.create` +checks from the machine that it reaches this Store, so no call comes +between `store.connect` and the first Sandbox. `machines` holds only the +machines with a connection of their own (H9); an entry without `url` and +`password` (a record written while machines were attached one by one, +2026-11-12 to 2026-11-17) is read as none, and its `checked` is dropped at +the record's next write. `machine.remove` forgets the machine's entry in +every Store. Nothing converts a sqlite Store into PostgreSQL: a project reaches PostgreSQL only by an Init into it. ### Credentials diff --git a/docs/architecture/engine/store/postgresql.md b/docs/architecture/engine/store/postgresql.md index 3372f9326a..c9bd97b48f 100644 --- a/docs/architecture/engine/store/postgresql.md +++ b/docs/architecture/engine/store/postgresql.md @@ -240,4 +240,4 @@ nothing. - [Store](README.md): the seam, admission, history calls and errors. - [SQLite](sqlite.md): the tables these mirror. - [Diagnostics](diagnostics.md): the PostgreSQL audit and its physical checks. -- [Control: Stores and projects](../../control/stores-and-projects/sqlite-and-postgresql.md): how a home connects and attaches a PostgreSQL Store. +- [Control: Stores and projects](../../control/stores-and-projects/sqlite-and-postgresql.md): how a home connects, updates and checks a PostgreSQL Store. diff --git a/docs/architecture/flows/call.md b/docs/architecture/flows/call.md index d8c0127106..5ed33eb8a4 100644 --- a/docs/architecture/flows/call.md +++ b/docs/architecture/flows/call.md @@ -48,7 +48,7 @@ retries. A lost answer after the request was sent is `Unknown`, never resent. | # | What happens | Owning topic | | --- | --- | --- | | 1 | The front end makes one library call: the CLI parses `layerfs `; Node and Python pass JSON text. | [Front ends](../api/front-ends/README.md), [Bindings](../api/front-ends/bindings.md) | -| 2 | The call table finds `family.call` (85 calls in 12 families) and checks every argument before any effect. | [Calls](../api/calls/README.md#what-it-does), [SDK](../api/sdk/README.md) | +| 2 | The call table finds `family.call` (84 calls in 12 families) and checks every argument before any effect. | [Calls](../api/calls/README.md#what-it-does), [SDK](../api/sdk/README.md) | | 3 | A folder scope, when the client has one, guards and rewrites the checked arguments. | [Calls: folder scope](../api/calls/README.md#folder-scope) | | 4 | The home is opened (explicit path, else `LAYERFS_HOME`, else `~/.layerfs`). | [Home](../control/home/README.md) | | 5 | The family function reads the Store and project records, refuses a blocking intent and takes this process's lease on the Store. | [Stores and projects](../control/stores-and-projects/README.md#what-it-does), [Home: the `op` intent](../control/home/README.md#the-op-intent) | diff --git a/docs/architecture/glossary.md b/docs/architecture/glossary.md index bfe501bd15..28d08cf38d 100644 --- a/docs/architecture/glossary.md +++ b/docs/architecture/glossary.md @@ -18,7 +18,7 @@ LayerFS (backend, install, scope), the row gives both. | Branch | A named line of Commits in a project, with a base Layer and an optional head Commit. `main` is not a Branch. | [History: records](core/history/README.md#records) | | Bridge | `layerfs-bridge`: the wire contract between a controller and a Workspace daemon: the authenticated channel, every control and SDK record, the daemon's private setup and the Store install records. | [Bridge](engine/bridge/README.md) | | Bucket | A named place the home records where backups are kept, reached through a driver (local folder, MinIO, R2 or S3 API). Drivers know files by name and nothing about Stores. | [Backups: buckets](control/backups/buckets.md) | -| Call table | The 85 calls in 12 families that every front end exposes, with their arguments, and the one dispatcher they all go through. | [Calls](api/calls/README.md), [SDK](api/sdk/README.md) | +| Call table | The 84 calls in 12 families that every front end exposes, with their arguments, and the one dispatcher they all go through. | [Calls](api/calls/README.md), [SDK](api/sdk/README.md) | | Capture | The Commit step that seals a Workspace's active overlay generation as its published frontier and opens the next one. It copies no rows. | [Commit: what it does](engine/commit/README.md#what-it-does) | | Commit | An immutable history record: a complete filesystem root, its parent, the base Layer it was built from, an optional second parent and metadata. Also the call (`workspace.commit`) that makes one from a Workspace. | [History: records](core/history/README.md#records), [Commit](engine/commit/README.md) | | Construction | Building canonical file and tree objects from input and saving them: Init from a native folder, a Commit from its captured generation. A Commit has one construction producer. | [Commit: construction](engine/commit/construction.md), [Project Init: construction](engine/project-init/README.md#construction), [Files: construction](core/files/README.md#construction) | diff --git a/docs/versioned/2.0.0/README.md b/docs/versioned/2.0.0/README.md index 050e9e2ceb..2aa55a3042 100644 --- a/docs/versioned/2.0.0/README.md +++ b/docs/versioned/2.0.0/README.md @@ -25,7 +25,7 @@ Commit, merge, and publish to `main`. Stores: a sqlite Store is one file on one machine, a postgresql Store is a database. Backups of sqlite Stores go to r2 or MinIO buckets. See [machines, Stores and backups](machines-and-stores.md). -- **One call table.** 85 calls in twelve families: `home`, `project`, `branch`, +- **One call table.** 84 calls in twelve families: `home`, `project`, `branch`, `commit`, `file`, `sandbox`, `workspace`, `machine`, `store`, `backup`, `logs` and `doctor`. The `layerfs` command, the Node package and the Python package are thin @@ -43,7 +43,7 @@ Commit, merge, and publish to `main`. Python. 2. [The home](home.md): the layout of `LAYERFS_HOME`, the one-time conversion of a 1.1.0 home, and `layerfs import-env`. -3. [The calls](calls.md): all 85 calls with their positionals and options, +3. [The calls](calls.md): all 84 calls with their positionals and options, results, errors and exit codes, and the shell switches. 4. [Machines, Stores and backups](machines-and-stores.md): machine kinds, manual start and stop, sqlite and postgresql Stores, push and @@ -64,7 +64,7 @@ Commit, merge, and publish to `main`. The code wins over every document below where they differ. - [The call table](../../../api/crates/layerfs-api/src/), one file per call - under its family (`/.rs`): the 85 calls, their arguments and + under its family (`/.rs`): the 84 calls, their arguments and options. - [The #533 work order](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/533/README.md) and [its decisions log](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/533/DECISIONS.md): owner diff --git a/docs/versioned/2.0.0/calls.md b/docs/versioned/2.0.0/calls.md index e35bc66d7e..eebdcff416 100644 --- a/docs/versioned/2.0.0/calls.md +++ b/docs/versioned/2.0.0/calls.md @@ -9,7 +9,7 @@ one file per call under its family in help, and the Node and Python functions are all built from that table, so the three front ends take the same arguments. -There are 85 calls in twelve families. `layerfs help` lists them, and +There are 84 calls in twelve families. `layerfs help` lists them, and `layerfs help []` prints one family or one call. ## How a call is written @@ -629,18 +629,17 @@ $ layerfs machine status native See [machines](machines-and-stores.md#machines) for kinds, defaults and refusals. -### `store` (13) +### `store` (12) | Call | Positionals | Options | Does | | --- | --- | --- | --- | | `list` | — | — | every Store, from the home's records, each with its `state` | | `get` | `name` | — | a Store's record | -| `status` | `name` | — | a Store's live state | +| `status` | `name` | `machine` | a Store's live state; with `machine`, whether that machine reaches a PostgreSQL Store (`check`) | | `stats` | `name` | — | Store-wide publication and allocation counters | | `create` | `name` | — | an empty sqlite Store on `native` | -| `connect` | `name`, `url` | `password`, `synchronous_commit` | attaches a PostgreSQL database as a Store, attached to `native` | -| `attach` | `name`, `machine` | `url`, `password` | lets a machine run Sandboxes of a PostgreSQL Store, after checking from that machine that it reaches the Store's database and may write it | -| `detach` | `name`, `machine` | — | stops a machine running Sandboxes of a PostgreSQL Store; refused while one runs there | +| `connect` | `name`, `url` | `password`, `synchronous_commit` | connects a PostgreSQL database as a Store; its Sandboxes run on every machine that supports them | +| `update` | `name` | `machine`, `url`, `password`, `synchronous_commit`, `clear` | changes a PostgreSQL Store's connection; with `machine`, gives that machine a connection of its own (checked from it first), or removes it with `clear` | | `push` | `name`, `machine` | `on_progress` | sends a sqlite Store from `native` to a machine | | `pull` | `name` | `on_progress` | brings a sqlite Store from its machine to `native` | | `remove` | `name` | — | deletes a sqlite Store with every project in it | diff --git a/docs/versioned/2.0.0/machines-and-stores.md b/docs/versioned/2.0.0/machines-and-stores.md index ec3d7b7b3f..81bd3baf2f 100644 --- a/docs/versioned/2.0.0/machines-and-stores.md +++ b/docs/versioned/2.0.0/machines-and-stores.md @@ -453,39 +453,42 @@ layerfs project init pg-main/load ./load under the Store's lease. - PostgreSQL is built into every artifact, so no build flag selects it; the Store's connection string decides whether it is used. -- A PostgreSQL Store's Sandboxes run on the machines it is attached to. - `store connect` attaches `native` with the Store's own connection and - records its check (`machines.native.checked`) when the role may write; - `store attach ` attaches another machine, or attaches - one again with a new connection: +- A PostgreSQL Store is on no machine. Its Sandboxes run on every machine + whose platform runs PostgreSQL Sandboxes (`native` on macOS and Linux, + `docker`, `lima` and `apple-container`), with no call between `store + connect` and `sandbox create`: each machine dials the Store's URL and + password (its loopback host rewritten as below), and `sandbox create` + checks from the machine that the server answers and holds this Store. + `store update` changes the connection: ```bash # not run - layerfs store attach pg-main docker - layerfs store attach pg-main build-vm --url "postgresql://lfs@10.0.0.5/layerfs" --password file:/run/secrets/pg-vm - layerfs store detach pg-main docker + layerfs store update pg-main --synchronous-commit off + layerfs store update pg-main --url "postgresql://lfs@db.internal/layerfs" --password env:LAYERFS_PG + layerfs store update pg-main --machine build-vm --url "postgresql://lfs@10.0.0.5/layerfs" --password file:/run/secrets/pg-vm + layerfs store update pg-main --machine build-vm --clear + layerfs store status pg-main --machine docker ``` - Without `--url` and `--password` the machine dials the Store's URL and - password (its loopback host rewritten as below); with either, its own. - The check runs from the machine (in this process for `native`, else in - the machine's worker, which must be from a bundle with `store.reach`): - the server answers, the database holds the `layerfs` schema with the - Store's binding (another one is `Unavailable` "holds another Store"), - and the role may use the schema, make temporary tables and read and - write every LayerFS table and sequence (else `Denied`, naming the first - privilege missing). A refused check records nothing; a passed one - records the attachment with `checked: {at, server_version, - latency_ms}`. The record keeps the URL without a password and the - password as a reference, as `store connect` does. The platforms are - `native` (macOS and Linux), `docker`, `lima` and `apple-container`; an - attach to a machine the support table refuses is `Unsupported` with its - reason. `store detach` is `InUse` while a Sandbox of the Store is - recorded on that machine. `sandbox create` on a machine the Store is not - attached to is `Unsupported`, naming the `store attach` that fixes it; - `machine remove` is `InUse` while a PostgreSQL Store is attached to the - machine. `machine list` and `machine get` show a machine's attached - PostgreSQL Stores as `postgresql`. + Without `--machine` it sets the Store's own URL, password reference and + `synchronous_commit`; a new URL or password is dialled once from this + process and must reach this Store. With `--machine` it gives that machine + a connection of its own (`--url`, `--password`, either alone), checked + from the machine before it is saved; `--clear` removes it. The check + runs in this process for `native`, else in the machine's worker (which + must be from a bundle with `store.reach`): the server answers, the + database holds the `layerfs` schema with the Store's binding (another + one is `Unavailable` "holds another Store"), and the role may use the + schema, make temporary tables and read and write every LayerFS table and + sequence (else `Denied`, naming the first privilege missing). A refused + check saves nothing. `store status --machine ` runs the same + check on request and answers it as `check: {machine, host, ok, ..}`; it + records nothing. The record keeps the URL without a password and the + password as a reference, as `store connect` does, and lists under + `machines` only the machines with a connection of their own. A machine + the support table refuses is `Unsupported` with its reason. `machine + remove` forgets the machine's own connection. A running Sandbox keeps the + connection it started with. - A Sandbox starts where the machine starts Sandboxes (the daemon and the worker always build the PostgreSQL engine). The caller resolves the password and rewrites a loopback host for the machine (`docker`: diff --git a/docs/versioned/2.0.0/migration.md b/docs/versioned/2.0.0/migration.md index 617150cfb6..8da3dce3a6 100644 --- a/docs/versioned/2.0.0/migration.md +++ b/docs/versioned/2.0.0/migration.md @@ -37,7 +37,7 @@ Nothing here is published to npm, PyPI or crates.io. | Old | 2.0.0 | Status and reason | | --- | --- | --- | | npm `@layerfs/agent` (`import { LayerFS } from "@layerfs/agent"`) | npm `layerfs`: `import layerfs from "layerfs"` or `import { client, LayerFSError } from "layerfs"`. An ES module for Node 22 or later, MIT | **Replaced.** One package, one native addon. [`api/node/package.json`](../../../api/node/package.json), [Node](../../architecture/api/front-ends/bindings.md#layerfs-node) | -| npm `@layerfs/cli` (the Node `layerfs` command; #521: 106 commands) | the native `layerfs` command, one Rust binary, 85 calls plus `doctor`, `import-env`, `worker` and `help` | **Replaced.** It is not an npm package: `layerfs-cli` has `publish = false`. The command counts are not comparable (the old ones included the backend and runtime commands). [CLI](../../architecture/api/front-ends/README.md#the-layerfs-command) | +| npm `@layerfs/cli` (the Node `layerfs` command; #521: 106 commands) | the native `layerfs` command, one Rust binary, 84 calls plus `doctor`, `import-env`, `worker` and `help` | **Replaced.** It is not an npm package: `layerfs-cli` has `publish = false`. The command counts are not comparable (the old ones included the backend and runtime commands). [CLI](../../architecture/api/front-ends/README.md#the-layerfs-command) | | npm `@layerfs/backend` (`LayerFSBackend.of(client)`, 14 operator and diagnostic calls) | none, except `layerfs.workspace.status(ws)` | **Removed** (O25). [Limitations](limitations.md#not-built-refused-as-unsupported), [README](README.md#compatibility-with-110) | | npm `@layerfs/runtime` (`LayerFSRuntime.open`, worker start and restart, `project.start_init` / `get_init` / `subscribe_init`, Sandbox and volume sweep) | no package. The `machine`, `store`, `project` and `sandbox` families; `project.init` with `on_progress`; `machine start` and `machine stop` by hand (nothing starts a machine on demand, D11) | **Replaced** in part, **Removed** for the job registry (O22) and the sweep. [Machines and Stores](machines-and-stores.md#starting-and-stopping-manual-535-d11), [Node and Python](sdk.md#functions-as-options) | | npm `@layerfs/mcp` (#521: 50 tools, stdio) | none | **Removed.** MCP is shelved (O32): 2.0 has no MCP server, and this guide gives no MCP setup. [Coverage](../../architecture/api/calls/coverage.md#outside-the-table) | @@ -126,7 +126,7 @@ options. The argument lists are in the generated Calls with no counterpart in the 1.1.0 method table above: `project.delete`, `commit.get`, and the `machine`, `store` and `backup` families, plus the -commands `layerfs doctor` and `layerfs import-env`. In all there are 85 calls in +commands `layerfs doctor` and `layerfs import-env`. In all there are 84 calls in twelve families ([coverage](../../architecture/api/calls/coverage.md)). ## 4. Folder scope diff --git a/docs/versioned/2.0.0/sdk.md b/docs/versioned/2.0.0/sdk.md index 3b08d98492..3a5edf309c 100644 --- a/docs/versioned/2.0.0/sdk.md +++ b/docs/versioned/2.0.0/sdk.md @@ -136,7 +136,7 @@ asyncio.run(main()) See [`project.export`](../../architecture/api/sdk/project.md#projectexport) for destination validation, fidelity, routing, refusal codes and result fields. -The table has 85 calls, seven in the `project` family. +The table has 84 calls, seven in the `project` family. ```js // /exports exists; the destination is absent or empty on this host. diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/access.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/access.rs index 1acd60c367..ae56ca4e8d 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/access.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/access.rs @@ -1,4 +1,4 @@ -//! What a role may do in a Store's schema, for `store.attach`'s check of a +//! What a role may do in a Store's schema, for the check of a //! machine's connection and for a PostgreSQL compaction's preflight: the //! server's version, one round trip's latency, the role, and each privilege //! the Store's writes and compaction need that it lacks. One connection