From c731c536fb20263f0e1f73d64ee49f3ab374a763 Mon Sep 17 00:00:00 2001 From: Tequila Sunset Date: Sat, 8 Aug 2026 18:17:56 +0800 Subject: [PATCH] docs: define the 0.1.0 product contract --- AGENTS.md | 148 ++++++++++-------------- CONTEXT.md | 124 ++++++++++++++++++++ README.md | 29 +++-- ROADMAP.md | 75 ++++++++++--- design/README.md | 5 +- design/api-documentation.md | 6 +- design/architecture.md | 25 +++-- design/benchmarks.md | 8 +- design/cluster.md | 16 +-- design/codegen.md | 24 ++-- design/errors.md | 23 ++-- design/observability.md | 22 ++-- design/persistence.md | 111 +++++++++++------- design/release-0.1.0.md | 188 +++++++++++++++++++++++++++++++ design/release.md | 8 +- design/runtime.md | 51 +++++---- design/scheduling.md | 20 ++-- design/simulation.md | 44 ++++---- design/testing.md | 4 +- design/timers.md | 133 ++++++++++++++-------- design/transport.md | 6 +- docs/README.md | 12 +- docs/compatibility.md | 9 +- docs/errors.md | 3 +- docs/example.md | 18 ++- docs/programming-model.md | 218 +++++++++++++++++++++++++----------- docs/release-0.1.0.md | 154 +++++++++++++++++++++++++ docs/vision.md | 100 ++++++++++------- docs/writing-style.md | 85 ++++++++++++++ examples/shadow/README.md | 44 +++++--- 30 files changed, 1256 insertions(+), 457 deletions(-) create mode 100644 CONTEXT.md create mode 100644 design/release-0.1.0.md create mode 100644 docs/release-0.1.0.md create mode 100644 docs/writing-style.md diff --git a/AGENTS.md b/AGENTS.md index 97831da..5c18816 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,93 +1,59 @@ # Agent Instructions -## 项目状态 - -实现进度以 [ROADMAP.md](ROADMAP.md) 为准，不要从本文件或代码自行推断。 - -不要为了「让仓库看起来有东西」写占位代码、半成品或假实现。 - -## 环境 - -Go 在 `$HOME/sdk/go1.26.5/bin`，staticcheck 在 `$HOME/go/bin`。两个目录都不在默认 PATH 里。开始工作先执行： - -```bash -export PATH="$HOME/sdk/go1.26.5/bin:$HOME/go/bin:$PATH" -export GOPROXY=https://goproxy.cn -``` - -Go module proxy 必须走 `goproxy.cn`。`proxy.golang.org` 在这台机器上不通。 - -`/tmp` 是 tmpfs。任何 fsync 相关的测量放在 `/tmp` 里都是空转，数字作废。存储 benchmark 要把 `GOR_BENCH_DIR` 指到真盘。 - -## 仓库布局 - -``` -docs/ 产品 spec —— gor 该满足什么（产品语言 + 领域语言） -design/ 设计 spec —— 系统该怎么实现（可用技术语言） -research/ 实测证据 —— 支撑 design 决策的事实 -``` - -## 协作 - -每个 agent 在自己的 worktree 里干活。按批次推进，每批做完停下来报告，等评审通过再进下一批。 - -依赖安装（`go get` / `go mod tidy`）是 agent 的事，不要指望别人代劳。不要让两个 agent 同时写 `go.mod`。 - -评审意见和代码不一致时，直接说：「你说的和代码对不上」。不要把代码改成评审意见描述的样子。 - -提交前只检查和提交自己负责的文件。不要 push，不要开 PR。 - -进度以 [ROADMAP.md](ROADMAP.md) 为准，它是叙事 spec：讲「做什么、为什么」，是大步骤和它们的理由。GitHub 的 milestone + issue 是执行单元：讲「这件具体的事、现在什么状态、归哪个版本」。issue 不复述设计，只指向 ROADMAP 或 design 文档的对应小节。 - -## Spec 先行 - -先把方案写进 `docs/` 或 `design/`，再实现。实装追赶 spec，不是 spec 跟着实装走。文档里出现尚未实装的能力是正常的。 - -某篇文档与代码有显著差距时，在文内单列「差距」小节说明现状。**正文是 spec，差距是脚注。** - -`docs/` 禁止技术语言（包名、函数签名、存储表结构）。那些归 `design/`。 - -## 不可协商的约束 - -这几条来自 [design/testing.md](design/testing.md)，违反任何一条都会让确定性模拟测试整体失效——而 DST 是本项目的主要差异点： - -1. 所有 I/O 在接口后面。 -2. 时间通过注入的 `Clock` 获取。生产代码里出现 `time.Now()` 即视为 bug。 -3. 组件是显式状态机，状态转换是可枚举的函数。 -4. **跨调用的等待用 channel，不用 mutex。** mutex 阻塞在 `synctest` 里不算 durably blocking。这条连带禁用了 `x/sync/singleflight`。 - -ROADMAP 第 4 步（DST 骨架）必须在第 6 步（集群）之前。顺序不可协商——这四条约束无法事后加装。 - -## 测试 - -```bash -make test # 单元测试，单个 < 50ms，不起网络不起进程 -make sim # 确定性模拟测试，慢，不进默认 test -make gen # 生成器端到端测试，起 go list 子进程，不进默认 test -make net # 真 TCP 的传输测试，不进默认 test -make lint # vet + staticcheck -``` - -改完代码务必跑 `make ci`。 - -改动涉及 `runtime` / `cluster` / `store` 时，迭代过程中就单独跑 `make sim`，不要等到最后。 - -禁止：真实外部依赖、真实时间（`time.Sleep` 做同步、轮询墙钟做断言）、`t.Skip` 掩盖偶发失败、新旧测试并存。 - -需要 Go 1.25+（`testing/synctest` GA 版本）。 - -## 注释原则 - -默认不写注释；用命名、类型、函数边界让代码自解释。只有当代码无法表达「为什么这样做」时才写——外部系统限制、关键不变量、反直觉选择。 - -注释在解释「做什么/怎么做」，就重构代码让注释消失。 - -注释不引用设计文档、issue 或任务编号——文档会改名移动，引用必然腐烂。历史归 git log。 - -公开 API 的 doc comment 是使用者契约，不属于本节对实现注释的限制；它只写可依赖的行为，不叙述实现。 - -## 设计原则 - -模型尽可能简洁，只包括必要的属性。 - -不加保护性特判——越死板，特例越多。 +## Project Context + +- Treat [ROADMAP.md](ROADMAP.md) as the source of truth for implementation + progress. +- Treat [CONTEXT.md](CONTEXT.md) as the source of truth for product language. + Use its terms. +- Work within the current product and compatibility contracts. Do not add + placeholder, partial, fake, or speculative features. + +## Engineering Principles + +- Study established products before designing a solution. Reuse proven + patterns and conventions when they fit the current requirements. +- Choose the simplest design that fully meets the current requirements. +- Grow the system in working layers. Do not trade a working product for + unfinished complexity. +- Keep modules small and keep different concerns separate. +- Check existing dependencies before adding code or a package. Prefer a + maintained library when it reduces complexity or improves reliability. +- Make architecture decisions for the long term. Do not create a stopgap that + is meant to be replaced later. +- Remove obsolete paths. Add compatibility code only when the product contract + requires it. +- Keep models small. Add only the properties that the current contract needs. + +## Architecture Constraints + +- Follow the architectural constraints in [design/testing.md](design/testing.md) + when changing runtime, store, or cluster code. +- Keep all I/O behind interfaces. +- Get time from an injected `Clock`. +- Use explicit state machines for components with concurrent behavior. +- Use channels for waits across calls. Do not use mutexes for this purpose. +- Keep the deterministic simulation foundation ahead of new cluster work. + +## Documentation + +- Write or update the product or design spec before implementation. +- Use `docs/` for product requirements and user language. +- Use `design/` for technical design. +- Use `research/` for measured evidence. +- Write repository documents in the simple English defined by + [docs/writing-style.md](docs/writing-style.md). +- When a document differs from the implementation, add a clear `Gap` section. + +## Collaboration + +- Work in a separate worktree. +- Work in reviewed batches. Stop after each batch and report the result. +- Keep changes within the assigned files and preserve unrelated changes. +- Write commit messages in simple English. State the actual change. +- Before handoff, inspect the diff and report the verification result. + +## Verification + +- Before handoff, run the repository CI gate: `make ci`. +- Report failed tests, missing tools, and environment limits. Do not hide them. diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..9f5009e --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,124 @@ +# Orleans Grain Runtime + +This context defines the Orleans language used by gor. gor is a Go port of the +Orleans runtime model. Use these terms in product docs, design docs, examples, +and new code names. Keep one term for one concept. + +## Grain model + +**Grain**: +A stateful object identified by a GrainId. A Grain has state and behavior. +_Avoid_: entity, actor, service, worker + +**GrainId**: +The stable identity of one Grain. It contains a GrainType and a GrainKey. +_Avoid_: identity, address, instance ID, object ID + +**GrainType**: +The kind of Grain. Grains with different GrainTypes do not share a GrainId. +_Avoid_: class, model, category + +**GrainKey**: +The application value that selects one Grain of a GrainType. +_Avoid_: ID, name, identifier + +**Grain Reference**: +A typed value that names a Grain without creating it. +_Avoid_: proxy, handle, stub, pointer + +**Call**: +A request to run one method on a Grain through a Grain Reference. +_Avoid_: message, invocation, packet + +**Request Context**: +Small data attached to a Call. The called Grain can read it during the Call. +Request Context is not State and the Grain Runtime does not save it. +_Avoid_: header, context value, request property + +**Call Filter**: +Shared policy that runs before or after a Call. +_Avoid_: interceptor, middleware + +## Runtime model + +**Grain Runtime**: +The part of the application that starts Grains, accepts Calls, keeps State, +and runs Reminders. +_Avoid_: engine, server, actor system, framework + +**Activation**: +The live form of a Grain that can receive Calls. A Grain can lose its +Activation without losing its GrainId or State. +_Avoid_: instance, process, actor + +**Deactivation**: +The end of an Activation. Deactivation does not delete a Grain's GrainId or +State. +_Avoid_: destroy, delete, terminate + +**Lifecycle**: +The path from Activation to Deactivation for one Grain. +_Avoid_: object lifetime, process lifetime + +## State and reminders + +**State**: +The current data owned by a Grain. State describes the Grain now. +_Avoid_: status, condition + +**Confirmed State**: +State that the Grain Runtime has accepted as the current value for a Grain. +_Avoid_: saved state, cached state, best-effort state + +**Durability**: +The amount of Confirmed State that remains after a machine failure. +_Avoid_: speed mode, flush mode + +**Reminder**: +A future Call that the Grain Runtime remembers for a Grain. A Reminder can +happen once or repeat on a period. +_Avoid_: timer, wake-up, scheduled task, job + +## Call results + +**Conflict**: +A result that says a write used old State. A newer State already exists. +_Avoid_: collision, race error, stale write error + +**Unknown Result**: +A result where the caller cannot know if a Business Action happened. A timeout +or delivery failure can cause an Unknown Result. +_Avoid_: failed call, lost call, partial success + +**Business Action**: +A change that the application asks a Grain to make. +_Avoid_: side effect, operation, command + +**Safe Repeat**: +A Business Action that does not apply the same business change twice when it +runs more than once. +_Avoid_: idempotent action, exactly-once action + +## Boundaries + +**Application**: +The program that defines Grains and their business rules. +_Avoid_: client, consumer, user code + +**Silo**: +A process that hosts a Grain Runtime and its Activations. In 0.1.0, one Silo +runs on one machine. +_Avoid_: node, machine, server, worker + +**Single Silo**: +One Silo with local State. This is the main gor product in 0.1.0. +_Avoid_: standalone mode, local cluster + +**Cluster**: +Several Silos that share Grain ownership. Cluster support is an optional +extension. +_Avoid_: multi-node runtime, shared service + +**Ownership**: +The Silo responsible for serving a Grain in a Cluster. +_Avoid_: placement, assignment, shard owner diff --git a/README.md b/README.md index 2cfb730..9ac3969 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,24 @@ # gor -**A persistent, stateful runtime for Go.** A single binary, library-shaped, embeddable, designed for deterministic simulation testing from day one. +**A persistent Grain Runtime for Go.** It is embeddable, runs as one Silo, +and is designed for deterministic simulation testing. -> **Status:** single-process features are implemented and usable. Multi-node calls can be routed and forwarded to the node that owns the entity; neighbor failure is decided by direct probing and death voting; errors carry stable codes across nodes. Detailed progress: [ROADMAP.md](ROADMAP.md) +> **Status:** single-Silo features are implemented and usable. Cluster +> features are an optional preview. Detailed progress: [ROADMAP.md](ROADMAP.md) ## What this is -A Go library that makes objects with an identity, state, single-threaded execution, and crash recovery your programming unit. You write ordinary Go interfaces and ordinary structs; `gor` handles activation, call serialization, persistence, and scheduled wake-ups. Cross-node distribution is an optional extension, not the main line: it exists for workloads that have outgrown one machine, and single-node users are not asked to pay for it. +A Go library that makes a Grain with a GrainId, State, serialized Calls, and +restart recovery your programming unit. You write Go interfaces and structs. +`gor` handles Activation, Call ordering, persistence, and Reminders. A +future cluster is an optional extension, not the main line. -The idea comes from Microsoft Orleans' virtual actor model, but this is not a port of Orleans. The trade-offs are recorded one by one in the [ADR and design documents](design/README.md); the three most important: +`gor` is a Go port of the Orleans runtime model. The Go API uses Go forms, +but the product terms and runtime meaning follow Orleans. The main design +rules are in the [design documents](design/README.md): - The programming model is typed at compile time, not `any` in, `any` out — proxies are generated from Go interfaces ([design/codegen.md](design/codegen.md)). -- Single-node is a first-class citizen, not a degenerate mode of clustering. No sidecar, no external database — `import` it and it works. +- One Silo is a first-class product. It needs no sidecar or remote service. - Deterministic simulation testing is an architectural constraint, not a testing technique retrofitted afterwards ([design/testing.md](design/testing.md)). This is the main difference between this project and comparable implementations. ## Why it exists @@ -29,13 +36,13 @@ Measured details: [research/landscape.md](research/landscape.md) (in Chinese). ## What it does not do -`gor` explicitly does not pursue these; the reasons are in [docs/vision.md](docs/vision.md): +`gor` does not provide these in 0.1.0; the boundaries are in +[docs/vision.md](docs/vision.md): -- No Orleans API compatibility layer, and no one-to-one correspondence of concepts. -- No general-purpose actor framework (no supervision trees, mailbox policies, or behavior switching — the Akka-style capabilities). -- No workflow DSL or orchestration graphs. -- No "unbounded horizontal scaling". The target scale is a single machine to a small cluster. -- No cross-entity transactions. A call that touches two entities and fails halfway fails halfway — `gor` gives no rollback and no outbox. If you need atomicity, make them one entity. +- No source or binary compatibility promise with Orleans. +- No Call Filters. +- No reentrant or interleaved Grain Calls. +- No cluster operation tools or unbounded scale. ## Documentation diff --git a/ROADMAP.md b/ROADMAP.md index 451f127..eef087b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -2,13 +2,15 @@ **Direction.** `gor` is for programs that need stateful objects on one machine. The main line is the single-node product; clustering is an optional extension that exists and is shipped, but is not the path the project is built around. See [docs/vision.md](docs/vision.md). -**Status.** The single-node core (steps 1–5.5) is implemented and usable; for practical purposes, the single-node product is done. Clustering (step 6) is implemented and shipped in 0.0.x as a preview. Every item under "Required before an announced release" is done. Staying in 0.0.x rather than announcing is a choice the project has already made — see [design/release.md](design/release.md); it is a posture, not a roadmap step. +**Status.** The single-node core (steps 1–5.5) is implemented and usable; for practical purposes, the single-node product is done. Clustering (step 6) is implemented and shipped in 0.0.x as a preview. The pre-announcement checklist is done. The announced 0.1.0 target and its remaining composition and failure-evidence work are defined below. Slicing principle: every step runs, is accepted, and delivers value on its own. Dependencies of the form "step 1 cannot be verified until step 6" are not allowed. ## The single-node core -Steps 1 through 5.5 form the single-node product: a per-key runtime, persistent state, typed proxies, the simulation harness, scheduled tasks, and the API fixes the first real example surfaced. They are implemented. +Steps 1 through 5.5 form the single-Silo product: a per-Grain runtime, +persistent State, typed Grain References, the simulation harness, Reminders, +and the API fixes the first real example surfaced. They are implemented. ### 1. Single-process runtime @@ -50,15 +52,17 @@ Node crashes require `runtime` to gain a stop path that does not drain. This is **Acceptance:** a fixed seed reproduces a sequence of injected store faults and node crashes, and a rerun yields a byte-identical decision sequence; invariants hold under those faults; a double activation created by two nodes sharing one store is blocked by the ETag on write conflict instead of being silently overwritten. -What is reproduced is the injected decisions, not the whole execution. A fault deactivates the activation, and from then on even the entity's value depends on scheduling — see [design/simulation.md](design/simulation.md). +What is reproduced is the injected decisions, not the whole execution. A +fault deactivates the Activation, and later Grain State can depend on +scheduling — see [design/simulation.md](design/simulation.md). -### 5. Scheduled tasks +### 5. Reminders One table plus one poller. Design: [design/timers.md](design/timers.md). Deliberately not repeating Orleans Reminders v1's design — the in-memory cache plus ring-partitioning scheme is what Orleans itself replaced in v2 (`Orleans.DurableJobs`). Start directly with a table plus a poller. -The scheduled-task table does not go through `store.Store`; it is a new interface. It is also a new fault source on the step-4 skeleton — it must be hooked up in this step, not deferred to step 6. +The Reminder table does not go through `store.Store`; it is a new interface. It is also a new fault source on the step-4 skeleton — it must be hooked up in this step, not deferred to step 6. Claiming via CAS must be done right now. In step 6, two nodes' pollers can scan the same row at the same time; fixing it then would mean rewriting all the earlier tests. @@ -71,7 +75,8 @@ Implemented. The example's factory now takes only `*gor.Binder`; the load genera The friction from writing the first real example was minor, but all of it sat on the main path: - `gor.Now(b)` — the Binder already holds the injected `Clock`; without it, users would write `time.Now()`. -- `gor.Ref[T](b, key)` — an entity calling another entity should not require the factory to capture a runtime object. +- `gor.Ref[T](b, key)` — a Grain calling another Grain should not require the + factory to capture a Runtime object. - `OnError` — scheduled delivery failures used to be dropped silently; they are now visible to users through the unified error sink. - `OnActivate` / `OnDeactivate` — the lifecycle hooks used to be missing; they are now implemented as optional interfaces, so the example can be notified on activation and eviction. @@ -81,13 +86,24 @@ Placed before step 6 because it changes the public API. API changes get more exp ## The single-node line, going forward -The single-node core above is, for practical purposes, done. A user who runs `gor` on one node has the whole product: typed entities, state that survives a crash, calls serialized per key, scheduled tasks that survive a restart, lifecycle hooks, observability, and a stable error contract. What follows is not "making single-node usable" — it already is. The two steps below — a durability control for state writes and the reproducible-test foundation — are implemented. Clustering (step 6, below) is parked and was never a prerequisite for either. +The single-Silo core above is usable. A user who runs `gor` on one machine +has typed Grains, State that survives a crash, serialized Calls, Reminders, +lifecycle hooks, observability, and a stable error contract. The 0.1.0 work +still has to make these capabilities one public experience. Clustering is +optional and is not a prerequisite. ### A durability control for state writes -A state write is the operation single-node users care about most — it bounds how many state changes one entity can do per second ([design/benchmarks.md](design/benchmarks.md)). The write path gets a durability control: a way to run writes at a looser tier when the application accepts the trade. The tier is chosen at store open, applies to entity state only, and defaults to Full; the exact option, what a crash can cost at each tier, and the migration of older databases are in [design/persistence.md](design/persistence.md). This step is the capability and its measured baseline, not a redesign of the store interface. +A State write is the operation single-Silo users care about most. It bounds +how many State changes one Grain can make per second +([design/benchmarks.md](design/benchmarks.md)). The durability control is +implemented and applies to Grain State only. Its exact limits are in +[design/persistence.md](design/persistence.md). -Implemented. `store.WithDurability` on either SQLite constructor selects the tier — `DurabilityFull` (the default) or `DurabilityRelaxed`. Entity-state rows live in a database file derived from the named path by inserting `-state` before the extension; the schedule and membership tables stay in the named file and always run at Full. The Relaxed tier flushes the state database's write-ahead log on Close, and a database from an earlier 0.0.x migrates into the new layout on first open. The baseline is recorded at both tiers on real disk: Full 1.7 ms/op, Relaxed 14 us/op ([benchmarks.md](benchmarks.md)). +Implemented. `store.WithDurability` selects Full or Relaxed durability for +Grain State. Reminder and membership data stays at Full durability. The +baseline is recorded at both tiers on real disk: Full 1.7 ms/op, Relaxed 14 +us/op ([benchmarks.md](benchmarks.md)). **Acceptance.** A single-node user can pick a durability tier without writing their own store; the benchmark records a number at the relaxed tier alongside the full-durability baseline; the durability trade is stated in product language in the docs. @@ -103,6 +119,27 @@ Implemented. The decision encoding reads driver-owned liveness only — `simulat bbolt and pebble are candidates for a single-node store, and a single-node-first store raises their relevance: the design leaned toward SQLite partly for cluster reasons — SQLite "satisfies both state storage and coordination tables", and coordination tables are a cluster need ([design/persistence.md](design/persistence.md)). Postgres was cluster-only and leaves with clustering. This is a goal, not a gap: the store interface is public, a user can ship their own backend, and no measured `gor`-specific number shows bbolt or pebble beating a relaxed-durability SQLite for this workload. The durability control above is the step with evidence; this becomes a step only when measurement shows a real user pain. They stay deliberately un-milestoned goals. +## Announced release target: 0.1.0 + +The next announced release is governed by the [0.1.0 product contract](docs/release-0.1.0.md) and delivered in the order specified by [design/release-0.1.0.md](design/release-0.1.0.md). This is a target specification, not a claim that the work is complete. + +The release target keeps the product single-Silo and makes the Grain, State, +Reminder, lifecycle, Call, persistence, and observability capabilities +dependable as one public experience. Cluster production is outside this +target. A release item is complete only when its failure behavior is tested +and the conformance Application can use it through the public API. + +The implementation batches are: + +1. Freeze the public contract and acceptance matrix. +2. Harden Grain Activation, Calls, lifecycle, shutdown, and errors. +3. Verify State and Reminder recovery under crashes and duplicate attempts. +4. Complete call context, serialization, and application-storage boundaries. +5. Run the conformance application and deterministic failure suite. +6. Pass clean-install and full repository release gates. + +The first batch is documentation-only and must be reviewed before implementation begins. + ## Optional extension: clustering Clustering is implemented and shipped in the 0.0.x tags. It is an optional extension, not the main line: it exists for workloads that have outgrown one machine, and single-node users are not asked to pay for it. The boundary is stated in [docs/vision.md](docs/vision.md) and in user terms in [docs/programming-model.md](docs/programming-model.md): during the window while nodes disagree about ownership, a write that always succeeds on a single node can return a conflict to the caller, who must retry. Further cluster work — rolling upgrades and operational cleanup — is deliberately deferred; it is not on the main line. @@ -115,13 +152,15 @@ This step must state plainly, in both docs and API: the directory is eventually Too big; sliced into four segments. The split points are chosen on "does it need the network" — the dividing line is transport, then probing. -6t depends on none of the earlier segments and can run in parallel with 6a: it only deals with operating-system sockets and knows nothing of entities, identities, or the membership table. +6t depends on none of the earlier segments and can run in parallel with 6a: +it only deals with operating-system sockets and knows nothing of Grains, +GrainIds, or the membership table. #### 6a. Membership table and ring A membership table, a node state machine (joining / active / dead), view polling, a hash ring, and local routing decisions. No transport: if the computed target is not this node, return an error carrying the owner's address. -A new table and a new fault source, shaped like step 5's scheduled-task table — deliberately so; step 5 just blazed this trail. +A new table and a new fault source, shaped like step 5's Reminder table — deliberately so; step 5 just blazed this trail. 6a's membership-table-and-ring stage only defines member states and the view; the evidence for declaring death is completed by 6c's probe voting. @@ -133,7 +172,8 @@ Implemented. The two boundaries of declaring death are written into [design/clus A thin, self-written transport: long-lived connections, multiplexing, frames, lazy dialing. Design: [design/transport.md](design/transport.md). -It knows nothing of entities, identities, or the membership table — it moves bytes. So it does not depend on 6a and can proceed in parallel. Implemented. +It knows nothing of Grains, GrainIds, or the membership table. It moves +bytes, so it does not depend on 6a. Implemented. **Acceptance:** out-of-order responses match their requests; a response that arrives after a timeout is dropped, not handed to the next request; when a connection breaks, every in-flight request returns with an error and no goroutine leaks; an oversized frame does not make the peer allocate memory based on the frame header. @@ -141,7 +181,10 @@ It knows nothing of entities, identities, or the membership table — it moves b Wire 6a's routing decisions to 6t's transport, plus a fake network (delay, packet loss, partition). Envelope and forwarding semantics: the "Forwarding" section of [design/cluster.md](design/cluster.md); how the server side recovers types from bytes: [design/codegen.md](design/codegen.md). -Implemented. Calls to entities not on this node are forwarded through the transport to the node that currently owns them, sharing the same call path as local calls; the fake network deterministically simulates partitions, drops, recovery, and delay. Reorder is not a distinct fault under this transport model (see [design/simulation.md](design/simulation.md)). Probing and death voting are 6c. +Implemented. Calls to Grains not on this Silo are forwarded through the +transport to the Silo that owns them. Local and forwarded Calls share one +call path. The fake network simulates partitions, drops, recovery, and +delay. Probing and death voting are 6c. This step already changed the generated artifacts. `Invoke`'s argument went from `[]any` to `any`; each method has one request struct, and each type has one constructor like `newAccountCall`. Like step 3 taking over `dispatch`, this is a planned breaking change. @@ -161,11 +204,11 @@ The boundary: the two sides of a partition can vote each other dead, even to the ## Required before an announced release -Not part of any step above, but completed before gor points users at a version. All items below are done; gor currently sits at 0.0.x (publicly visible tags, not announced — see [design/release.md](design/release.md)). Whether and when to announce a version is a maintainer judgment, and the current choice is not to announce; it is not a roadmap step. +Not part of any step above, this checklist is the baseline that was completed before the 0.1.0 target was formed. gor currently sits at 0.0.x (publicly visible tags, not announced — see [design/release.md](design/release.md)). The additional 0.1.0 composition, failure-evidence, and usability requirements are tracked in [docs/release-0.1.0.md](docs/release-0.1.0.md). -- ~~English documentation. Done last — the docs are still changing; translating early means translating twice.~~ **Done.** `README`, `ROADMAP`, `FINDINGS`, `benchmarks.md`, the six `docs/` files, `examples/shadow/README`, and all 17 `design/` files are now English-only, the Chinese originals fully replaced with nothing kept in both languages; `research/`, `AGENTS.md`, `CLAUDE.md`, and `.github/PULL_REQUEST_TEMPLATE.md` stay in Chinese as internal evidence and maintainer-facing text — commits and reviews are written in Chinese anyway — and every link to `research/` carries an `(in Chinese)` marker. +- ~~English documentation. Done last — the docs are still changing; translating early means translating twice.~~ **Done.** `README`, `ROADMAP`, `FINDINGS`, `benchmarks.md`, the six `docs/` files, `examples/shadow/README`, and all 17 `design/` files are now English-only, the Chinese originals fully replaced with nothing kept in both languages; `research/`, `CLAUDE.md`, and `.github/PULL_REQUEST_TEMPLATE.md` stay in Chinese as internal evidence and maintainer-facing text — commits and reviews are written in Chinese anyway — and every link to `research/` carries an `(in Chinese)` marker. - ~~Public API doc comments. To be completed after step 6c, once the public API is finalized as a release candidate; must meet [design/api-documentation.md](design/api-documentation.md) before `v0.1.0`.~~ **Done.** -- ~~Error and cancellation contract. Stable error codes and the cross-node cancellation boundary must be implemented before `v0.1.0`.~~ **Done.** Spec: [docs/errors.md](docs/errors.md) and [design/errors.md](design/errors.md). The stable code is the only cross-node identity of an error; the cancellation boundary is implemented per spec. The spec previously had one self-contradiction (merged errors matched locally but not across nodes); it was ruled that "the error code is the unique reachable value in the error tree", and the implementation was brought in line. +- ~~Error and cancellation contract. Stable error codes and the cross-node cancellation boundary must be implemented before `v0.1.0`.~~ **Done.** Spec: [docs/errors.md](docs/errors.md) and [design/errors.md](design/errors.md). A stable code is the only cross-node error identifier. The cancellation boundary is implemented per spec. - ~~Root runtime shutdown contract. The spec is complete, see [design/runtime.md](design/runtime.md), [design/cluster.md](design/cluster.md), and [docs/programming-model.md](docs/programming-model.md); implementation had not started. Before `v0.1.0`, new calls must stop being admitted during the shutdown window.~~ **Done.** The root runtime's stop state machine and single admission gate are implemented: four transition functions, atomic `admit`/release; the public `Invoke` / inbound handler / scheduled delivery share one gate that sits before the ownership decision and forwarding; `closing→killing` is an escalation, not a no-op; cluster nodes explicitly report their end reason via `DeclaredDead()`; stop coordination uses receive channels only. Transport teardown comes after admitted forwarded requests and inbound replies. Also fixed a real bug where a declared-dead node sent an empty view and triggered a graceful deactivation. - ~~Deactivation reasons and the background error sink for lifecycle hooks. The spec is complete, see [design/runtime.md](design/runtime.md), [design/timers.md](design/timers.md), and [docs/programming-model.md](docs/programming-model.md); the hooks themselves were implemented, but the deactivation reasons (`DeactivationReason`) and the structured background error sink (`BackgroundError`) were not — both are public API breaking changes. These two must be delivered before `v0.1.0`.~~ **Done.** `OnDeactivate` receives the deactivation reason (idle, ownership lost, normal shutdown, instance untrusted); the reason is fixed at the first transition out of the active state, and the hook gets a work context with no deadline that is never canceled; the background error sink now emits events whose sources are a closed set — scheduled delivery carries the method name, deactivation hook failure carries the deactivation reason, nothing outside the package can add sources, and sources are no longer guessed from method names. Poller scan and claim failures and deliveries canceled mid-shutdown do not enter the sink. Two public API migrations ship with this item (`OnDeactivate` gains a parameter, `OnError` takes an event). - ~~A real example application, rerun with the new signatures after step 5.5~~ **Done.** See [examples/shadow/](examples/shadow/); design: [docs/example.md](docs/example.md). Its output is [FINDINGS.md](FINDINGS.md) — nine API frictions; the first six went into step 5.5, the README's non-goals, or doc additions; the last three record frictions that still exist. diff --git a/design/README.md b/design/README.md index feaecf0..973087d 100644 --- a/design/README.md +++ b/design/README.md @@ -2,7 +2,7 @@ This layer describes **how the system should be implemented**: architectural boundaries, data models, interfaces, technology choices, and trade-offs. -Technical language is allowed. The division of labor with [`docs/`](../docs/README.md): `docs/` says what must be satisfied; `design/` says how. +Use the ASD-STE100 writing rules in [`docs/writing-style.md`](../docs/writing-style.md). Use the core terms in [`CONTEXT.md`](../CONTEXT.md). Technical terms are allowed when they make the design clear. Define a term when a new reader may not know it. The division of labor with [`docs/`](../docs/README.md): `docs/` says what must be satisfied; `design/` says how. ## Annotation conventions @@ -16,7 +16,7 @@ When a document diverges significantly from the code, it lists a "Gap" section i - [runtime.md](runtime.md) — activation, directory, lifecycle. - [scheduling.md](scheduling.md) — serial execution, reentrancy, mailbox. - [persistence.md](persistence.md) — state storage, CAS, backend choice. -- [timers.md](timers.md) — persisted scheduled tasks: the table, the poller, delivery semantics. +- [timers.md](timers.md) — persisted Reminders: the table, the poller, delivery semantics. - [cluster.md](cluster.md) — membership, placement, directory consistency. - [transport.md](transport.md) — byte transport between nodes: frames, connections, multiplexing, and close semantics; the substrate boundary for forwarding. - [errors.md](errors.md) — stable error codes, the call error envelope, and the cross-node cancellation boundary. @@ -27,6 +27,7 @@ When a document diverges significantly from the code, it lists a "Gap" section i - [benchmarks.md](benchmarks.md) — what the performance baseline measures and does not, and the measurement conditions comparable numbers must carry. - [api-documentation.md](api-documentation.md) — English doc comments for the public API: contract boundaries, scope, example trade-offs, and the acceptance process. - [release.md](release.md) — version numbers, release thresholds, the manual release checklist, and how release-note blocks are handled. +- [release-0.1.0.md](release-0.1.0.md) — the implementation order, failure matrix, conformance example, and evidence gates for the first announced release. ## Decision records diff --git a/design/api-documentation.md b/design/api-documentation.md index c774904..9abf596 100644 --- a/design/api-documentation.md +++ b/design/api-documentation.md @@ -69,7 +69,7 @@ A doc comment is not a second user manual. The table below gives each kind of in | Content | Home | | --- | --- | -| The full mental model of entities, identities, calls, state, and scheduled tasks; the complete path of cross-entity calls | `docs/programming-model.md` | +| The full mental model of Grains, GrainIds, calls, state, and Reminders; the complete path of cross-Grain calls | `docs/programming-model.md` | | What may be relied on, version upgrades, and breaking changes | `docs/compatibility.md` and release notes | | Activation cache, mailbox, CAS tables, polling, membership table, frame format, algorithm trade-offs | The relevant `design/` document | | Multi-node limitations, protocol details, the full matrix of configuration combinations | `docs/` or the relevant `design/` document | @@ -82,7 +82,7 @@ A comment may state one local restriction; it must not restate a whole model jus gor adds no Go `Example` functions and sets none as a release gate. -A realistic root-package call must at least define an entity interface, generate proxies, build a runtime, install the artifacts, register factories, and then obtain a reference and call. It is not a lightweight example demonstrating one declaration. Inside an `Example`, it would compile and run on every `make test`; any v0 change to artifact shape, startup order, or public signatures would mean maintaining a third call path, plus text results when `Output` is present. `docs/example.md` and `examples/shadow/` already carry the full path; duplicating it would not improve v0.1's contract clarity. +A realistic root-package call must at least define a Grain interface, generate proxies, build a runtime, install the artifacts, register factories, and then obtain a reference and call. It is not a lightweight example demonstrating one declaration. Inside an `Example`, it would compile and run on every `make test`; any v0 change to artifact shape, startup order, or public signatures would mean maintaining a third call path, plus text results when `Output` is present. `docs/example.md` and `examples/shadow/` already carry the full path; duplicating it would not improve v0.1's contract clarity. Later, an `Example` is added only for a stable scenario that a local comment cannot explain and that genuinely deserves to run directly on pkg.go.dev. Before adding, all of the following must hold: no dependence on real time, network, or processes; no hidden generation step; runs within the default test constraints; its output expresses a stable observable contract. Otherwise keep it as a documentation snippet or an example application; do not create a test entry that rots. @@ -99,4 +99,4 @@ From then on, the same change that adds or alters a supported declaration must u ## Gap -The candidate API documentation batch is in place: the supported packages (`gor`, `clock`, `store`, `transport`, and `cmd/gorgen`) carry package docs and per-symbol contracts on their independently usable entry points, and the implementation packages (`runtime`, `mail`, `timer`, `cluster`) carry package docs stating the no-direct-dependency boundary. The manual review this section's acceptance step 2 prescribes has been performed: the root `Activation` alias keeps no independent comment because `Activations()` documents the aggregate as a sorted snapshot of the runtime's active entities, the `Identity` field is a documented type, and `Queued` is a plain count of calls awaiting dispatch with no separate default, failure, concurrency, or lifecycle rule — the aggregate exception applies. No Go `Example` functions exist, as prescribed. +The candidate API documentation batch is in place: the supported packages (`gor`, `clock`, `store`, `transport`, and `cmd/gorgen`) carry package docs and per-symbol contracts on their independently usable entry points, and the implementation packages (`runtime`, `mail`, `timer`, `cluster`) carry package docs stating the no-direct-dependency boundary. The manual review this section's acceptance step 2 prescribes has been performed: the root `Activation` alias keeps no independent comment because `Activations()` documents the aggregate as a sorted snapshot of the runtime's active Grains, the `GrainId` field is a documented type, and `Queued` is a plain count of calls awaiting dispatch with no separate default, failure, concurrency, or lifecycle rule — the aggregate exception applies. No Go `Example` functions exist, as prescribed. diff --git a/design/architecture.md b/design/architecture.md index d4a50a6..2047655 100644 --- a/design/architecture.md +++ b/design/architecture.md @@ -25,11 +25,13 @@ gor ──────────▶ transport Dependencies point only downward. `runtime` does not know `cluster` exists, and **does not need to leave any interface for it**. -[Step 6b](../ROADMAP.md#6b-forwarding)'s routing happens in the `gor` layer: every call first asks the ring who owns this Identity; if it is self, hand to `runtime`; if someone else, forward (see [cluster.md](cluster.md)). `runtime`'s interface does not change one word — it is about "calls on the same key are serialized", unrelated to why this key lands on this node. +[Step 6b](../ROADMAP.md#6b-forwarding)'s routing happens in the `gor` layer: every call first asks the ring who owns this GrainId; if it is self, hand to `runtime`; if someone else, forward (see [cluster.md](cluster.md)). `runtime`'s interface does not change one word — it is about "calls on the same key are serialized", unrelated to why this key lands on this node. -`runtime` also does not import `store`: entity state is read and written by `gor` inside the factory closure; `runtime` only hands out an Identity and gets back an opaque instance (see [persistence.md](persistence.md)). +`runtime` also does not import `store`: Grain State is read and written by +`gor` inside the factory closure; `runtime` only hands out a GrainId and gets +back an opaque instance (see [persistence.md](persistence.md)). -The only thing `runtime` gains because of the cluster is an entry to drop activations by Identity: after a view change, `gor` uses it to drop entities that no longer belong to this node. It shares the idle-eviction path and does not reveal the cluster's existence. +The only thing `runtime` gains because of the cluster is an entry to drop activations by GrainId: after a view change, `gor` uses it to drop Grains that no longer belong to this node. It shares the idle-eviction path and does not reveal the cluster's existence. **Single-node mode injects nothing**; `gor` takes the local branch directly. No fake implementation that always returns this node is built to "leave an extension point" — that is living dead code; not one line of it is needed in steps 1 through 5. @@ -39,10 +41,10 @@ The only thing `runtime` gains because of the cluster is an entry to drop activa | --- | --- | --- | | `gor` | Public API, configuration assembly | Any algorithm | | `runtime` | Activation cache, lifecycle, local directory, request dispatch | Network, storage implementations | -| `mail` | The serial execution queue of a single entity | Knowing what an entity is | -| `store` | State read/write plus the CAS table abstraction and its backends | Knowing entity semantics | -| `timer` | Scan due, claim, deliver (see [timers.md](timers.md)) | Knowing entity semantics | -| `cluster` | Membership table, node state machine, view polling, consistent-hash ring (see [cluster.md](cluster.md)) | Executing entity methods, forwarding | +| `mail` | The serial execution queue of a single Grain | Knowing what a Grain is | +| `store` | State read/write plus the CAS table abstraction and its backends | Knowing Grain semantics | +| `timer` | Scan due, claim, deliver (see [timers.md](timers.md)) | Knowing Grain semantics | +| `cluster` | Membership table, node state machine, view polling, consistent-hash ring (see [cluster.md](cluster.md)) | Executing Grain methods, forwarding | | `transport` | Byte transport between nodes | The semantics of the encoding format | | `sim` | Fake network, fake clock, fault injection, invariant assertions | Production code paths | | `cmd/gorgen` | The code generator | Runtime behavior | @@ -63,7 +65,7 @@ Orleans has 30k lines of serialization code, mostly for version tolerance (old a The cost is explicit: **rolling upgrades to incompatible method signatures without downtime are not supported.** The gain is an entire subsystem not built. -`encoding/json` is chosen, the same story as entity state persistence. The reason is no second serialization story, plus humans can read it directly in production — worth more than performance when debugging cross-node problems. +`encoding/json` is chosen, the same story as Grain state persistence. The reason is no second serialization story, plus humans can read it directly in production — worth more than performance when debugging cross-node problems. **No `Codec` interface.** An interface with one implementation is ceremony. When encoding really needs to change, the change is in the few encode/decode lines, not in the shape of an interface. @@ -71,10 +73,13 @@ Encoding happens in the `gor` layer. `transport` moves opaque bytes and does not ## Structural comparison with Orleans -No one-to-one correspondence is pursued, but the source of the size difference is recorded. Orleans' `src/` measures 274k lines, of which only about 26k genuinely need rebuilding on the Go side (directory + membership + activation + placement + hash ring + scheduler), because: +The Grain model follows Orleans. The Go implementation does not copy all +Orleans source. The source of the size difference is recorded. Orleans' +`src/` measures 274k lines, of which only about 26k need rebuilding on the Go +side (directory, membership, Activation, placement, hash ring, and Reminder +scheduler), because: - Serialization (31k lines) — not done, as said above. -- Streams, transactions, event sourcing, journaling (35k lines) — out of scope. - Cloud provider code (about 26k lines) — replaced by the embedded store and Postgres backends. - The `src/api/` baseline snapshot (35k lines) — not implementation code at all. diff --git a/design/benchmarks.md b/design/benchmarks.md index a42466d..45602d0 100644 --- a/design/benchmarks.md +++ b/design/benchmarks.md @@ -10,13 +10,13 @@ Two purposes; neither is about pretty numbers. ## Three measurements -**Invocation round trip** — in-memory store, single process, serial calls to a method that does nothing on one entity. It measures the runtime's own overhead: mailbox in and out, activation lookup, reflection dispatch. Mix in storage and it cannot be measured. +**Invocation round trip** — in-memory store, single process, serial calls to a method that does nothing on one Grain. It measures the runtime's own overhead: mailbox in and out, activation lookup, reflection dispatch. Mix in storage and it cannot be measured. -**State write** — real disk; how long one `Set()` takes to land. This is the number users care about most, because it decides how many state changes per second one entity can do. This number is recorded at each durability tier the store offers ([persistence.md](persistence.md)); the throughput gap between tiers is the only reason a tier exists, and a single number would hide it. Both are measured on real disk — on tmpfs the sync that separates the tiers is a no-op, so both tiers measure the same fake-fast number and the comparison is void. +**State write** — real disk; how long one `Set()` takes to land. This is the number users care about most, because it decides how many state changes per second one Grain can do. This number is recorded at each durability tier the store offers ([persistence.md](persistence.md)); the throughput gap between tiers is the only reason a tier exists, and a single number would hide it. Both are measured on real disk — on tmpfs the sync that separates the tiers is a no-op, so both tiers measure the same fake-fast number and the comparison is void. -**Cold activation** — after an entity is evicted, how long the first call takes. The core promise of virtual entities is "you do not manage the lifecycle"; this number states the price of that sentence. +**Cold activation** — after a Grain is evicted, how long the first call takes. The core promise of virtual Grains is "you do not manage the lifecycle"; this number states the price of that sentence. -This one needs a real-disk store, and the entity must carry state that was written before. Being evicted means the state went back to disk, and reading it back is the bulk of this cost. An in-memory store would measure the runtime's lookup-and-construct overhead, not the time the user waits. +This one needs a real-disk store, and the Grain must carry state that was written before. Being evicted means the state went back to disk, and reading it back is the bulk of this cost. An in-memory store would measure the runtime's lookup-and-construct overhead, not the time the user waits. Each of the three gets its own benchmark; no composite score. A composite score hides exactly the only useful information: which layer is slow. diff --git a/design/cluster.md b/design/cluster.md index 11b7216..a07171b 100644 --- a/design/cluster.md +++ b/design/cluster.md @@ -54,7 +54,7 @@ joining → active → dead **Declaring death**: decided only by the current neighbors' unexpired `suspect_votes`. A stale `iam_alive_at` is not evidence of death. -`dead` is terminal. A declared-dead node must not modify its own row even if it is still alive — its CAS fails on the etag mismatch, and then it must self-terminate; it must not keep serving under an identity the whole world considers dead. +`dead` is terminal. A declared-dead node must not modify its own row even if it is still alive — its CAS fails on the etag mismatch, and then it must self-terminate; it must not keep serving under a GrainId the whole world considers dead. **But a CAS failure alone is not proof of your own death.** A heartbeat CAS collision has two causes: someone else changed the row to `dead`, or the previous heartbeat actually landed and only the reply was lost on the way — the latter advances the etag without the node knowing. Both causes give the same signal, and self-termination is irreversible, so on a collision the node must read the whole table again: if its row is `dead` it self-terminates; otherwise it takes the fresh etag and keeps heartbeating. @@ -76,7 +76,7 @@ Death must go through the table; only then does the matter have an answer everyo "Must not modify your own row" is not enough. A declared-dead node still holds several activations whose ETags are stale, while new calls were already routed to other nodes. -So when a node sees its current generation as `dead` in a successfully read snapshot, it must report the cause "declared dead externally" to the root runtime. The root runtime first stops admitting entity calls and closes the public stop signal, then follows the abrupt stop: cancel executing calls, reject the queue, and drop activations. Calls after that return an error that the node has stopped serving. In the view a dead node computes, it owns nothing — but rejecting local calls cannot wait for the view to change; the old view may still assign some identity to it for a while. +So when a node sees its current generation as `dead` in a successfully read snapshot, it must report the cause "declared dead externally" to the root runtime. The root runtime first stops admitting Grain calls and closes the public stop signal, then follows the abrupt stop: cancel executing calls, reject the queue, and drop activations. Calls after that return an error that the node has stopped serving. In the view a dead node computes, it owns nothing — but rejecting local calls cannot wait for the view to change; the old view may still assign some GrainId to it for a while. An active `Close()` also writes the node's membership row as `dead`. That is only a normal leave — the root runtime already began a graceful stop — and the cluster node's completion signal must not be mistaken for an external death declaration. The cluster node must hand its end reason to the root runtime; a bare `Done` channel that carries no reason is not enough. @@ -104,7 +104,7 @@ This keeps the probe count per node constant at two. A bigger cluster does not i Probing reuses `Transport.Send` from [transport.md](transport.md). It goes through the same lazy dialing, framing, multiplexing, and fake-transport path. No separate UDP, HTTP, or side-channel sockets. -`cluster` does not import `transport`. It only depends on an async `Prober`: give it a target member ID, get back a reply channel. `gor`'s adapter sends the `probe` request defined in [Envelope](#envelope) via `Transport.Send`. The transport's server-side handler dispatches on `kind`; `probe` goes straight to `cluster`, not through an entity call. +`cluster` does not import `transport`. It only depends on an async `Prober`: give it a target member ID, get back a reply channel. `gor`'s adapter sends the `probe` request defined in [Envelope](#envelope) via `Transport.Send`. The transport's server-side handler dispatches on `kind`; `probe` goes straight to `cluster`, not through a Grain call. A probe request carries only `kind`. The server's current member ID goes into the ordinary response's `reply`, and the initiator compares it with the target in its snapshot; only an exact match counts as success. A new process reusing the address must not erase votes for an old generation. @@ -215,7 +215,7 @@ All of the following are verified in `make sim` with the fake transport and the ## Placement -A consistent-hash ring: nodes hash onto the ring by address, and entities land on the first `active` node by hashing their Identity. +A consistent-hash ring: nodes hash onto the ring by address, and Grains land on the first `active` node by hashing their GrainId. A hash ring is chosen over "random placement plus directory lookup": **a hash ring makes locating mostly pure local computation, with no network round trip.** The cost is that node changes cause activation migration. @@ -231,7 +231,7 @@ A virtual point's position comes from `hash(address + generation + index)`. Incl ### The ring is the directory; there is no second table -Placement is computed from `hash(Identity)` plus the current membership view — one pure local computation. **No separate directory table records "who is where".** +Placement is computed from `hash(GrainId)` plus the current membership view — one pure local computation. **No separate directory table records "who is where".** Orleans has a directory table because it does not place by hash: it puts activations on chosen silos and uses the ring only to partition the directory, so something must keep the books. `gor` places by ring; the ring itself is the ledger. @@ -259,7 +259,7 @@ Rejecting the directory table above is admitting this window cannot be closed: a ## Routing -Every call first computes which node `hash(Identity)` lands on: +Every call first computes which node `hash(GrainId)` lands on: - **Self** — hand to `runtime` as usual, exactly as in single-process mode. - **Someone else** — forward it (transport in the next section). @@ -268,7 +268,7 @@ Every call first computes which node `hash(Identity)` lands on: The ring and the membership view get their own package, shaped like `timer`: it takes a membership-table interface, a `Clock`, and its own address, and periodically reads the full table against the injected clock to compute the view. `gor` wires it up and, on view changes, hands the activations that no longer belong to this node to `runtime` for dropping. -**The ring is a pure function.** Give it a membership view and an Identity, and it computes a node. It reads no time, does no I/O, holds no state; unit tests feed it views directly. Fetching the view is the stateful half, kept separate from the ring. +**The ring is a pure function.** Give it a membership view and a GrainId, and it computes a node. It reads no time, does no I/O, holds no state; unit tests feed it views directly. Fetching the view is the stateful half, kept separate from the ring. ## Transport @@ -312,7 +312,7 @@ When the caller's ctx is canceled, the forwarding side drops the pending request ### Forwarding does not retry -Cannot send, connection dropped, the other side rejected — the error goes straight to the caller. Only the user knows whether retrying is safe; this is the same stance as with `State.Set()` conflicts and scheduled delivery failures. +Cannot send, connection dropped, the other side rejected — the error goes straight to the caller. Only the user knows whether retrying is safe; this is the same stance as with `State.Set()` conflicts and Reminder delivery failures. ## Migration diff --git a/design/codegen.md b/design/codegen.md index 89f8551..051aae8 100644 --- a/design/codegen.md +++ b/design/codegen.md @@ -11,7 +11,7 @@ acct := gor.Ref[Account](rt, "alice") // returns Account cannot be done with generics alone. The common solution in similar Go projects is to drop the types: ```go -resp, err := system.AskGrain(ctx, identity, msg, timeout) // any in, any out +resp, err := system.AskGrain(ctx, grainID, msg, timeout) // any in, any out ``` goakt does exactly this (measured: `AskGrain(ctx, *GrainIdentity, message any, timeout) (any, error)`, `GrainContext.Message() any` / `Response(any)`). The cost is that all type errors are deferred to runtime. @@ -20,7 +20,7 @@ goakt does exactly this (measured: `AskGrain(ctx, *GrainIdentity, message any, t ## The input contract -The generator reads user-written Go interfaces. A valid entity interface method must: +The generator reads user-written Go interfaces. A valid Grain interface method must: - take `context.Context` as its first parameter - return `error` last @@ -43,7 +43,7 @@ This contract comes from `alecthomas/go-rpcgen`'s approach (interface + named re Only marked ones are generated: ```go -//gor:entity +//gor:grain type Account interface { ... } ``` @@ -57,7 +57,7 @@ Each interface gets one generated proxy: ```go type accountProxy struct { - id gor.Identity + id gor.GrainId rt gor.Invoker } @@ -89,7 +89,7 @@ The generated file imports packages by their declared package name. When a metho An alias is the concatenation of the package path's trailing segments, sanitized into an identifier, extended one segment deeper until it is unique among the run's names: `a/domain` → `adomain`, `billing/domain/v2` → `v2`, `a/x/domain` and `b/x/domain` → `axdomain` and `bxdomain`. A numeric suffix (`domain2`) is the last resort. Assignment is deterministic — the same input always produces the same aliases — so regenerating does not churn the file. -The source package's own import line participates in the same allocation. When its name is one of the reserved names (`context`, `fmt`, `gor`), it collides with the generated file's fixed imports and is aliased like any other colliding import — an entity package `context` at `billing/context` imports as `billingcontext "billing/context"` — and every reference to the entity package in the generated file uses that alias. The source package keeps its name when it collides with nothing; a signature import that shares the source package's name is aliased away instead, never the other way around. +The source package's own import line participates in the same allocation. When its name is one of the reserved names (`context`, `fmt`, `gor`), it collides with the generated file's fixed imports and is aliased like any other colliding import — a Grain package `context` at `billing/context` imports as `billingcontext "billing/context"` — and every reference to the Grain package in the generated file uses that alias. The source package keeps its name when it collides with nothing; a signature import that shares the source package's name is aliased away instead, never the other way around. ## How generated artifacts plug into the runtime @@ -124,7 +124,7 @@ func newAccountCall(method string) (args any, reply any) It builds a pair of empty shells by method name: `"Deposit"` yields `&accountDepositRequest{}` and `&accountDepositReply{}`. An unrecognized method name yields nil for both — something that really happens between nodes on mismatched versions. -**The name carries the type, like `dispatchAccount` and `newAccountProxy`.** A package can hold several entity interfaces; a `newCall` without the type name would not compile once there is a second one. Everything in the artifacts that is generated per type carries the type in its name; no exceptions. +**The name carries the type, like `dispatchAccount` and `newAccountProxy`.** A package can hold several Grain interfaces; a `newCall` without the type name would not compile once there is a second one. Everything in the artifacts that is generated per type carries the type in its name; no exceptions. From here on it is all existing machinery: `json.Unmarshal` fills the args, they go through **the same `Invoke`**, and the result comes back as `json.Marshal(reply)`. From this point, forwarded calls and calls initiated by local proxies share one path; serialization, activation, and dispatch are not duplicated. @@ -138,7 +138,7 @@ Generated artifacts depend on one narrow interface: ```go type Invoker interface { - Invoke(ctx context.Context, id Identity, method string, args any, reply any) error + Invoke(ctx context.Context, id GrainId, method string, args any, reply any) error } ``` @@ -152,7 +152,7 @@ Load packages with `golang.org/x/tools/go/packages`, get type information from ` **A known pitfall**: `go/types` requires the loaded package to pass type checking. If the artifacts lived in the same package as the user interface, then "the artifacts do not exist yet" → "user code references them → the package fails type checking" → "the generator cannot load the package" — a deadlock. -The solution: **the artifacts land in their own package**. The entity package does not import them; `gor.Register` / `gor.Ref` connect them at runtime through the registry. That package must also be importable from the startup code that calls `Install`, so it is not `internal` — by default it sits at `/gorgen` (see [Invocation](#invocation)). +The solution: **the artifacts land in their own package**. The Grain package does not import them; `gor.Register` / `gor.Ref` connect them at runtime through the registry. That package must also be importable from the startup code that calls `Install`, so it is not `internal` — by default it sits at `/gorgen` (see [Invocation](#invocation)). ## How the generator is tested @@ -182,7 +182,7 @@ Generate after creating or changing a marked interface: go tool gorgen -pkg ./domain ``` -The output is a non-`internal` subpackage of the entity package — `/gorgen` — so the startup code can import it and call `Install`. `-out` picks another directory; the package name is always `gorgen`. `//go:generate` works too. +The output is a non-`internal` subpackage of the Grain package — `/gorgen` — so the startup code can import it and call `Install`. `-out` picks another directory; the package name is always `gorgen`. `//go:generate` works too. Why a separate `tool` line rather than plain `go run`: the generator depends on `golang.org/x/tools`, which the library never imports, so `go get` of the library alone leaves it out of `go.sum`. Splitting `cmd/gorgen` into its own module would fix the same thing, but it would force `internal/codegen` to leave `internal/`; the `tool` directive keeps the generator in-tree. @@ -194,7 +194,11 @@ No generation runs on `go build`: Go has no such hook, and forcing one would mak **Import aliases are assigned by the generator.** Two packages with the same name at different paths (`a/domain` and `b/domain`) in one method signature no longer produce code that does not compile; the generator aliases the colliding imports itself. The rule lives in "Import names" above. -**Scheduled-method handles are not generated.** A typed handle for `Schedule.Set` is a Go method expression on the entity interface, built by a hand-written `gor.Handle` in the root package; the generator emits nothing for it. Schedules are set from inside entity methods, and the entity package cannot import the package generated from its own interfaces (it would be a cycle: the generated package already imports the entity package for the interface types in its proxies and dispatch). A generated handle symbol could not be named from the code that sets a schedule, so the handle uses Go's method expressions instead. See [timers.md](timers.md). +**Reminder-method handles are not generated.** A typed handle for +`Reminder.Set` is a Go method expression on the Grain interface, built by +`gor.Handle`; the generator emits nothing for it. Reminders are set inside +Grain methods. The Grain package cannot import its generated package because +that would create an import cycle. See [timers.md](timers.md). ## Rejected approaches diff --git a/design/errors.md b/design/errors.md index 1619aec..d3a85c2 100644 --- a/design/errors.md +++ b/design/errors.md @@ -44,18 +44,18 @@ This version's framework code set is sealed as follows: | --- | --- | | `gor.no_owner` | The current view has no routable owner. | | `gor.node_dead` | The target node has stopped serving. | -| `gor.runtime_closed` | The runtime or the entity's mailbox is closed. | +| `gor.runtime_closed` | The runtime or the Grain's mailbox is closed. | | `gor.overloaded` | The call was rejected for a full queue before the method started. | -| `gor.type_not_installed` | The target node does not have this entity type. | +| `gor.type_not_installed` | The target node does not have this Grain type. | | `gor.unknown_method` | The target type has no such method. | | `gor.invalid_request` | The request's shape or arguments cannot be decoded under the current contract. | | `gor.persistence_conflict` | The state write hit a version conflict. | | `gor.persistence_failed` | The state write failed, and it was not a version conflict. | -| `gor.panic` | The factory or the entity method panicked. | +| `gor.panic` | The factory or the Grain method panicked. | | `gor.request_encode_failed` | The source could not encode the arguments into a call request. | | `gor.reply_encode_failed` | The return values of a successful call could not be encoded. | | `gor.transport_failed` | The request, response, or connection failed to transfer; the execution outcome is unknown. | -| `gor.call_cycle` | A call targeted an entity already occupied by the same call chain, so it could never start. | +| `gor.call_cycle` | A Call targeted a Grain already occupied by the same Call chain, so it could never start. | The framework must not invent `gor.*` codes outside this set for the same outcome. Applications must not use `gor.*`. This version registers no extra mappings for arbitrary error types and derives no codes from error text. @@ -92,9 +92,15 @@ Request-encoding failures, `Send` failures, and response-decoding failures also Local calls do not go through the envelope. They keep the original error object. As long as the error chain declares a `Code`, local `errors.Is` matches it by Go's standard rules. -Remote calls project the error on the server and rebuild it on the source. Projection uses `CodeOf`; rebuilding uses an error that matches only by code. The determinate code is therefore the one error identity both locations share. Text may add context but must not affect any branch. +Remote Calls project the error on the server and rebuild it on the source. +Projection uses `CodeOf`; rebuilding uses an error that matches only by code. +The stable code is the one error identifier both locations share. Text may add +context but must not affect any branch. -The framework constructs the table's codes on every public call path, including local ones. Then `errors.Is` does not depend on call location for framework codes either. Internal packages' old sentinels may remain as internal implementation details, but must not be the sole identity of a `gor` public call result. +The framework constructs the table's codes on every public Call path, +including local ones. Then `errors.Is` does not depend on Call location for +framework codes. Internal sentinels may remain implementation details, but +must not be the sole identifier of a `gor` public Call result. ## Cancellation @@ -112,7 +118,10 @@ The caller has no observable "delivered" boundary. It cannot conclude from cance ## What is not done -No arbitrary type registration, no field serialization, no fidelity of error chains or joined structures, no error-code codegen annotations, no cancellation frames, no remote deadline propagation. All of them widen the wire contract without changing the stable code, the one cross-node error identity. Code reachability takes the unique code through a join, but that is not fidelity: the joined members, their count, and their individual texts are not preserved across nodes. +No arbitrary type registration, field serialization, error-chain fidelity, +error-code codegen annotations, cancellation frames, or remote deadline +propagation. These features would widen the wire contract without changing +the stable error code. ## Gap diff --git a/design/observability.md b/design/observability.md index ac9b908..38bb35b 100644 --- a/design/observability.md +++ b/design/observability.md @@ -4,7 +4,7 @@ `gor` is responsible for providing a minimal set of runtime observability facts. -Application-side proxies can count calls but cannot see the activation directory or mailboxes. They cannot reliably answer how many activations exist, nor find entities with a backlog. Only the runtime holds these facts. +Application-side proxies can count calls but cannot see the activation directory or mailboxes. They cannot reliably answer how many activations exist, nor find Grains with a backlog. Only the runtime holds these facts. `gor` is not responsible for aggregation, storage, export, or alerting. Those depend on the monitoring system the application already has. Building them into the library would pull in dependencies and decide labels, retention, and sampling policy for the user. @@ -16,7 +16,7 @@ Only two kinds of facts are exposed. ```go type Activation struct { - Identity Identity + GrainId GrainId Queued int } @@ -25,7 +25,7 @@ func (rt *Runtime) Activations() []Activation `Activations` returns the activations on this node in the `active` state. Instances being created, deactivating, or already stopped are not in the result. -The result is sorted by `(Identity.Type, Identity.Key)`. One call returns a copy of one point in time. It is not retained and does not refresh itself. +The result is sorted by `(GrainId.Type, GrainId.Key)`. One call returns a copy of one point in time. It is not retained and does not refresh itself. `len(rt.Activations())` answers how many activations exist right now. `Queued` answers whose mailbox is backing up. Comparing it with the runtime's configured capacity tells how far from overload rejection you are. @@ -33,13 +33,13 @@ The result is sorted by `(Identity.Type, Identity.Key)`. One call returns a copy The snapshot observes only this node. In cluster mode, each node collects on its own; cross-node aggregation belongs to the application's monitoring system. -Activation time, last-used time, the executing method, and per-entity cumulative counts are not exposed. These values would let the runtime make no new decision, yet they grow state, lock contention, and label cardinality. +Activation time, last-used time, the executing method, and per-Grain cumulative counts are not exposed. These values would let the runtime make no new decision, yet they grow state, lock contention, and label cardinality. ### Call completion ```go type CallObservation struct { - EntityType string + GrainType string Method string Duration time.Duration Err error @@ -50,13 +50,19 @@ func OnCall(func(CallObservation)) Option `OnCall` follows the configuration shape of `OnError`. It is a callback, not an exporter interface. There is exactly one action here; inventing a single-method interface for it has no value. -A call made through an entity proxy or `Runtime.Invoke` fires the callback once, after the outcome is settled and before returning to the caller. `Duration` spans from entering the runtime to the settled outcome, including routing, activation, queuing, and method execution, not the callback itself. `Err` is the same error this call returns to the caller. +A call made through a Grain proxy or `Runtime.Invoke` fires the callback once, after the outcome is settled and before returning to the caller. `Duration` spans from entering the runtime to the settled outcome, including routing, activation, queuing, and method execution, not the callback itself. `Err` is the same error this call returns to the caller. -Applications choose metric dimensions with `EntityType` and `Method`, record latency distributions with `Duration`, and compute error rates with `Err != nil`. The entity key is not in the event. Using unbounded keys as metric labels lets the monitoring system run away; investigating a single entity's backlog should use the activation snapshot. +Applications choose metric dimensions with `GrainType` and `Method`, record +latency distributions with `Duration`, and compute error rates with +`Err != nil`. The GrainKey is not in the event. Using unbounded keys as metric +labels lets the monitoring system run away; use the Activation snapshot to +inspect one Grain's backlog. After caller cancellation, the callback still fires, with `Err` being the cancellation error. A canceled method may keep executing, but no second completion event is emitted. The event describes the outcome the caller saw; it does not pretend to know the final business outcome. -An entity method delivered by a scheduled task is an ordinary call and produces this event. `OnDeactivate` is not a call and produces none; its failures still go only through `OnError`. +A Grain method delivered by a Reminder is an ordinary Call and produces this +event. `OnDeactivate` is not a Call and produces none; its failures still go +only through `OnError`. When a forwarded call completes, the originating node records one end-to-end call. The receiving node must not record the same logical call again. Inbound forwarded calls still go through the same local execution path; no second dispatch semantics are set up. diff --git a/design/persistence.md b/design/persistence.md index 5637e27..e8aeb62 100644 --- a/design/persistence.md +++ b/design/persistence.md @@ -4,8 +4,8 @@ The `store` package does two different things; don't mix them: -1. **Entity state** — read and write one state per Identity, with optimistic concurrency. -2. **Cluster coordination tables** — the shared tables membership and scheduled tasks need, with CAS required. +1. **Grain state** — read and write one state per GrainId, with optimistic concurrency. +2. **Cluster coordination tables** — the shared tables membership and Reminders need, with CAS required. Item 2 is the foundation of `cluster` correctness (see [cluster.md](cluster.md)); item 1 only serves business logic. @@ -13,8 +13,8 @@ Item 2 is the foundation of `cluster` correctness (see [cluster.md](cluster.md)) ```go type Store interface { - Read(ctx context.Context, id Identity) (Record, error) - Write(ctx context.Context, id Identity, data []byte, expect ETag) (ETag, error) + Read(ctx context.Context, id GrainId) (Record, error) + Write(ctx context.Context, id GrainId, data []byte, expect ETag) (ETag, error) } type Record struct { @@ -27,7 +27,10 @@ type Record struct { An empty record (first access) is represented by the zero ETag; passing the zero ETag to `Write` means "require that this record currently does not exist". -**A missing record is not an error.** `Read` on a record that does not exist returns the zero `Record` and a nil error. "Does not exist" and "exists but is empty" are the same thing here — an entity's state is the zero value at first activation anyway, so call sites need not distinguish the two. +**A missing record is not an error.** `Read` on a record that does not exist +returns the zero `Record` and a nil error. The State layer still distinguishes +three cases for each named State value: the value is absent, the value is +present with the type's zero value, or the value is present with other data. ## ETag is not optional @@ -66,7 +69,7 @@ The current code provides only the in-memory and SQLite backends. bbolt, pebble, ## How State connects to the runtime -`gor.State[T]` needs to know which Identity it belongs to, which store to write, and what the current ETag is. In the user's struct it is just a field, and the factory `func() Account { return &account{} }` has nowhere to hand these to it. +`gor.State[T]` needs to know which GrainId it belongs to, which store to write, and what the current ETag is. In the user's struct it is just a field, and the factory `func() Account { return &account{} }` has nowhere to hand these to it. The solution is for the factory to take one more parameter: @@ -76,19 +79,39 @@ gor.Register[Account](rt, func(b *gor.Binder) Account { }) ``` -`NewState` registers the cell on the binder; when the runtime activates an entity, it reads the store once according to the registrations and distributes the values to the cells. +`NewState` registers the State value on the Binder. When the Runtime +activates a Grain, it reads the store once and distributes named values to the +State cells. -Identity comes from here too: +Each State value has four operations: ```go -func Self(b *Binder) Identity +Get() T +Exists() bool +Set(ctx context.Context, value T) error +Clear(ctx context.Context) error ``` -The binder already holds the Identity — `State` needs it to locate the storage row, and `Schedule` needs it to write the table. `Self` just hands it to the user; it adds nothing. +`Exists` checks whether the named value is present. It does not compare the +value with the type's zero value. `Clear` removes the named value from the +record and writes the remaining values with the current ETag. A successful +clear leaves the Activation usable and makes `Exists` return false. A clear +conflict or unknown write result follows the same deactivation rule as +`Set`. -**It must be a value on the Binder, not entity state.** Storing its own key in state rolls back together with write conflicts, and in a double-activation window an activation would read a stale value — the entity would mistake its own identity. +GrainId comes from here too: -**Reflection-based struct field scanning and backfilling is rejected.** It would keep the factory as `func() Account` and save users one line, but at the cost of `unsafe` to write unexported fields — and users could not see how the field came alive. The saved line is not worth that price. +```go +func Self(b *Binder) GrainId +``` + +The binder already holds the GrainId — `State` needs it to locate the storage row, and `Reminder` needs it to write the table. `Self` just hands it to the user; it adds nothing. + +**It must be a value on the Binder, not Grain state.** Storing its own key in state rolls back together with write conflicts, and in a double-activation window an activation would read a stale value — the Grain would mistake its own GrainId. + +**Reflection-based struct field scanning and backfilling is rejected.** It +would keep the factory as `func() Account` and save one line. It would also +need `unsafe` for unexported fields and would hide how State becomes live. ### Two more things on the Binder @@ -98,7 +121,7 @@ Time: func Now(b *Binder) time.Time ``` -The binder already holds the injected `Clock` — `Schedule` needs it to compute due times. `Now` just hands it out. If it were not handed out, users would write `time.Now()` in their entities — and that is the very first thing this project forbids. +The binder already holds the injected `Clock` — `Reminder` needs it to compute due times. `Now` just hands it out. If it were not handed out, users would write `time.Now()` in their Grains — and that is the very first thing this project forbids. Calling others: @@ -108,58 +131,68 @@ type Scope interface{ /* sealed: only *Runtime and *Binder implement it */ } func Ref[T any](scope Scope, key string) T ``` -`Ref` originally took only a `*Runtime`, so an entity wanting to call another entity would need the factory closure to capture the runtime object as well, changing the factory signature from `func(b *Binder) T` to `func(rt *Runtime, b *Binder) T`. Cross-entity calls are the most common thing virtual entities do; it should not be the heaviest parameter on the signature. +`Ref` originally took only a `*Runtime`, so a Grain wanting to call another Grain would need the factory closure to capture the runtime object as well, changing the factory signature from `func(b *Binder) T` to `func(rt *Runtime, b *Binder) T`. Cross-Grain calls are the most common thing virtual Grains do; it should not be the heaviest parameter on the signature. -So the binder holds the runtime, `Ref` takes a sealed interface, and both sides share the name. Sealing — an unexported method in the interface — stops users from implementing it: it is not an extension point, just two shapes of "a place that can resolve entities". +So the binder holds the runtime, `Ref` takes a sealed interface, and both sides share the name. Sealing — an unexported method in the interface — stops users from implementing it: it is not an extension point, just two shapes of "a place that can resolve Grains". -**No third thing on the Binder.** It is the seam between entity and runtime; anything stuffed into the seam must first answer "what happens if it is not stuffed". +**No third thing on the Binder.** It is the seam between Grain and runtime; anything stuffed into the seam must first answer "what happens if it is not stuffed". ## runtime does not import store -`runtime` and `store` are siblings in the architecture diagram; neither imports the other. But the `Binder` must reach both sides: only `runtime` knows the Identity at activation time, and `Store` is injected when `gor` assembles the configuration. +`runtime` and `store` are siblings in the architecture diagram; neither imports the other. But the `Binder` must reach both sides: only `runtime` knows the GrainId at activation time, and `Store` is injected when `gor` assembles the configuration. The solution: the factory is called by `runtime` and provided by `gor`: ```go type Registration struct { - Factory func(context.Context, Identity) (any, error) + Factory func(context.Context, GrainId) (any, error) Dispatch Dispatch } ``` -`runtime` hands out the Identity and gets back an opaque instance. It does not know the instance carries a Binder, nor that construction read storage once. `gor` is the only package that sees both `runtime` and `store`; the conversion between the two Identity types happens in exactly this one place. +`runtime` hands out the GrainId and gets back an opaque instance. It does not know the instance carries a Binder, nor that construction read storage once. `gor` is the only package that sees both `runtime` and `store`; the conversion between the two GrainId types happens in exactly this one place. The factory can now return an error — reading storage during activation can fail. This merges with factory panic into the same path: the activation is not established, and the error returns to the caller. ## How conflicts get back to the runtime -When `Set()` hits `ErrConflict`, the activation must be deactivated — but `runtime` does not know `ErrConflict` and should not. +When `Set()` or `Clear()` hits `ErrConflict`, the Activation must be +deactivated. `runtime` does not know `ErrConflict` and should not. The error cannot be relied on to propagate up. User methods can perfectly well swallow it and return nil, by which time the cached ETag is stale. Deactivation must be independent of how user code handles the error. -So `Set()` immediately sets a flag on the Binder; `gor`'s dispatch wrapper checks it after every call and wraps the result into a shape `runtime` recognizes: +So `Set()` and `Clear()` immediately set a flag on the Binder. `gor`'s +dispatch wrapper checks it after every Call and wraps the result into a shape +`runtime` recognizes: ```go type Discard struct{ Err error } ``` -When `runtime` sees `Discard`, it deactivates the activation and returns `Err` unchanged to the caller. It only knows "the entity says it can no longer be used", not why. This shares the path with post-panic deactivation. +When `runtime` sees `Discard`, it deactivates the Activation and returns +`Err` unchanged to the caller. It only knows that the Grain is no longer safe +to use. This shares the path with post-panic deactivation. -## One record per entity +## One record per Grain -`Store.Read` returns one record per Identity, so all of an entity's `State` cells together encode into one record: a JSON object whose keys are the names given at `NewState`. +`Store.Read` returns one record per GrainId, so all of a Grain's `State` cells together encode into one record: a JSON object whose keys are the names given at `NewState`. This is why the names exist — with one cell the name is indeed redundant, but with a second cell something must distinguish them; a name-free special case for "only one cell" would only make the two shapes look different in storage. -Any cell's `Set()` rewrites the whole record. So the ETag is entity-level, not field-level — exactly the granularity double-activation protection needs: even if two activations modify different fields, one must hit a conflict. +Any State `Set()` or `Clear()` rewrites the whole record. The ETag is +Grain-level, not field-level. Even if two Activations modify different State +values, one must hit a conflict. -## Encoding entity state +## Encoding Grain state How user state becomes `[]byte`: JSON, not replaceable. The reason is not that JSON is fast; it is readable — when something goes wrong, you can look directly at what is in storage. -User-injected `Codec` is rejected. An entity's state is one record assembled from multiple cells; the outer container and the values inside the cells must use the same encoding. Making it pluggable has only two outcomes: either the outer layer is always JSON and only the cells go through the codec — a fake codec, since non-JSON bytes cannot fit into a JSON container — or the outer layer is `map[string][]byte`, and then JSON base64-encodes every value, killing readability, which was the entire reason for choosing JSON. Not worth paying that price for a knob nobody asked for. +User-injected `Codec` is rejected. A Grain's State is one record assembled +from multiple values. The outer container and the values must use one +encoding. A second codec would add a second storage model without a current +product need. Encoding for transport between nodes is a separate matter; see [architecture.md](architecture.md). @@ -167,7 +200,7 @@ No version-tolerant encoding (automatic compatibility when fields are added or r ## After a failed write -When `Set()`'s write returns any error, two things happen: +When a `Set()` or `Clear()` write returns an error, two things happen: 1. **The in-memory value stays unchanged.** Without confirmation of the write, a cell must not pretend the write succeeded — otherwise memory and storage diverge from then on, and the user reads a value that may not exist in storage at all. 2. **Deactivate the activation.** The next call reads from the store again, gets a fresh ETag, and continues. @@ -214,9 +247,9 @@ The product rests on "state survives a crash" being true without the user readin ### Where the tier is chosen -At store-open time, once, for the life of the store. One option; no per-write knob, no per-entity setting, no runtime switching. +At store-open time, once, for the life of the store. One option; no per-write knob, no per-Grain setting, no runtime switching. -This is the smallest model that fits the need. The choice does not change during a run — a service picks the trade at start and lives with it — so it is a property of the store, not of each write. A per-call or per-entity tier would push a durability decision into entity code that has no business making it, and would add a branch to the write hot path for a flexibility nobody asked for. +This is the smallest model that fits the need. The choice does not change during a run — a service picks the trade at start and lives with it — so it is a property of the store, not of each write. A per-call or per-Grain tier would push a durability decision into Grain code that has no business making it, and would add a branch to the write hot path for a flexibility nobody asked for. The tier is an option on the SQLite constructor; omitted means Full, the current behavior: @@ -226,13 +259,13 @@ db, err := store.OpenSQLite("data/gor.db", ) ``` -`Durability` and its two values (`DurabilityFull`, `DurabilityRelaxed`) live in the `store` package: every backend that implements `Store` shares one type, and the dependency direction (`gor` imports `store`, never the reverse) is what forces it there. Both SQLite constructors take the option — `OpenSQLite` and `OpenSQLiteWithClock` — so a cluster node, which opens with a clock for membership snapshots, sets the state tier the same way a single-node program does. The option does not add a second path to the API: the user names one database, and the store derives the location of any additional database file from it (see "What this means for the SQLite backend"). The in-memory store has no durability tier — it holds nothing across a crash by design, so the option does not apply to it; the tiers are a property of the on-disk backends only. The runtime, the entity, and the write path are unaware of the tier. `Store.Write`'s contract — write the bytes, return the new ETag — is identical at both tiers; only how hard the backend pushes the bytes to storage differs. +`Durability` and its two values (`DurabilityFull`, `DurabilityRelaxed`) live in the `store` package: every backend that implements `Store` shares one type, and the dependency direction (`gor` imports `store`, never the reverse) is what forces it there. Both SQLite constructors take the option — `OpenSQLite` and `OpenSQLiteWithClock` — so a cluster node, which opens with a clock for membership snapshots, sets the state tier the same way a single-node program does. The option does not add a second path to the API: the user names one database, and the store derives the location of any additional database file from it (see "What this means for the SQLite backend"). The in-memory store has no durability tier — it holds nothing across a crash by design, so the option does not apply to it; the tiers are a property of the on-disk backends only. The runtime, the Grain, and the write path are unaware of the tier. `Store.Write`'s contract — write the bytes, return the new ETag — is identical at both tiers; only how hard the backend pushes the bytes to storage differs. ### Scope: the state table only -The tier applies to the table that holds entity state. It does not apply to the coordination tables the runtime keeps alongside it in the same database: +The tier applies to the table that holds Grain state. It does not apply to the coordination tables the runtime keeps alongside it in the same database: -- **The scheduled-task table stays at Full.** Delivery promises at most one firing per due time ([timers.md](timers.md)), and that promise rests on the claim step — the row advance that marks a due time as taken — being on storage before delivery proceeds. If that claim could be lost to a hard crash, the due time would look untaken on restart and fire again: a duplicate, which is exactly what at-most-once forbids. The schedule table is not subject to the durability trade. +- **The Reminder table stays at Full.** Delivery promises at most one firing per due time ([timers.md](timers.md)), and that promise rests on the claim step — the row advance that marks a due time as taken — being on storage before delivery proceeds. If that claim could be lost to a hard crash, the due time would look untaken on restart and fire again: a duplicate, which is exactly what at-most-once forbids. The reminder table is not subject to the durability trade. - **The membership table stays at Full.** It exists only in cluster mode, and the cluster already treats the shared table as eventually consistent: a lost recent heartbeat, vote, or death declaration only slows reconvergence and breaks no correctness invariant ([cluster.md](cluster.md)). But the cluster is an optional, parked extension, and there is no measured reason to widen its failure envelope. Keeping it at Full leaves the cluster's story untouched. This is the product framing made precise: durability is a property the user trades for *their own* state, not for the bookkeeping that holds the runtime's guarantees together. @@ -243,15 +276,15 @@ The built-in store runs SQLite in WAL mode. The tier maps to SQLite's per-databa The mapping is safe to offer because of WAL mode specifically: in WAL mode, the relaxed sync level does not corrupt the database on power loss — only recent, un-checkpointed commits can be lost. That is exactly the Relaxed tier's stated semantics; the store is never left unreadable. This is also why the corruption-allowing tier was rejected above: offering it would require stepping outside WAL's safety, and the trade would stop being "lose recent writes." -Because SQLite's sync level is set per database file, not per table, running the state rows at a relaxed tier while keeping the coordination tables at Full means the implementation keeps them in separate database files (or otherwise isolates their sync settings). One shared database under one pragma cannot express "state rows relaxed, schedule rows full." The contract above — the state tier follows the user, the coordination tables are always Full — is what the implementation must satisfy; the file layout is the implementer's choice. +Because SQLite's sync level is set per database file, not per table, running the state rows at a relaxed tier while keeping the coordination tables at Full means the implementation keeps them in separate database files (or otherwise isolates their sync settings). One shared database under one pragma cannot express "state rows relaxed, reminder rows full." The contract above — the state tier follows the user, the coordination tables are always Full — is what the implementation must satisfy; the file layout is the implementer's choice. -Putting the tables in separate files does not break atomicity that existed before. State writes, schedule writes, and membership writes never shared a transaction: the schedule and membership tables have their own interfaces, and a state `Set` and a schedule `Set` are deliberately not atomic ([timers.md](timers.md)). The split changes where each table lives, not whether any two of them commit together. +Putting the tables in separate files does not break atomicity that existed before. State writes, reminder writes, and membership writes never shared a transaction: the reminder and membership tables have their own interfaces, and a state `Set` and a reminder `Set` are deliberately not atomic ([timers.md](timers.md)). The split changes where each table lives, not whether any two of them commit together. On disk the store may hold more than one database file, each carrying its own `-wal`/`-shm` sidecars. Backups and direct `sqlite3` inspection must cover every file the store creates, not just the path the user named — copying only the main file leaves the write-ahead log behind, and the recovered state is stale or torn. ### Old databases -An earlier 0.0.x database keeps the state, schedule, and membership tables in one database file. When the state tier calls for it, the state rows move to their own database, and such a database is read into the new layout on first open. The constraint is fixed, not optional: every confirmed state row must come through the move — nothing lost, and the store stays readable. This is not a new promise; it is the 0.0.x promise that a later 0.0.x reads state an earlier 0.0.x wrote, applied to the layout change ([compatibility.md](../docs/compatibility.md)). +An earlier 0.0.x database keeps the state, reminder, and membership tables in one database file. When the state tier calls for it, the state rows move to their own database, and such a database is read into the new layout on first open. The constraint is fixed, not optional: every confirmed state row must come through the move — nothing lost, and the store stays readable. This is not a new promise; it is the 0.0.x promise that a later 0.0.x reads state an earlier 0.0.x wrote, applied to the layout change ([compatibility.md](../docs/compatibility.md)). The migration is the store's job, done once on first open of an old database, not the user's; the user does not hand-move rows or convert formats. An interrupted migration must be redoable or resumable on the next open: the old database is not destroyed until the new one is complete, so a crash mid-move leaves the store recoverable. Whether the implementation keeps the single-file layout when the tier is Full and splits only at Relaxed, or splits uniformly regardless of tier, is the implementer's choice — but whichever it is, the constraint above holds: confirmed state survives the upgrade, and the user passes one path either way. @@ -273,10 +306,10 @@ What the simulation must and does cover is the storage seam's correctness-releva The state-write baseline is recorded at each tier, because a single number would hide the only thing the tier exists for. Both numbers are measured on real disk: on tmpfs the sync that separates the tiers is a no-op, so both tiers measure the same fake-fast number and the comparison is void ([benchmarks.md](benchmarks.md)). The relaxed number is expected to be materially below the full-durability baseline; the measurement records by how much. -## The scheduled task table +## The Reminder table ``` -schedule(entity_type, entity_key, name, method, due_at, interval, etag) +reminder(grain_type, grain_key, name, method, due_at, interval, etag) ``` -This table does not go through the `Store` interface — scanning due rows, CAS claiming, and row deletion do not fit into "read and write one state per Identity". It has its own interface; details in [timers.md](timers.md). +This table does not go through the `Store` interface — scanning due rows, CAS claiming, and row deletion do not fit into "read and write one state per GrainId". It has its own interface; details in [timers.md](timers.md). diff --git a/design/release-0.1.0.md b/design/release-0.1.0.md new file mode 100644 index 0000000..25eb8b1 --- /dev/null +++ b/design/release-0.1.0.md @@ -0,0 +1,188 @@ +# 0.1.0 Delivery Design + +This document lists the work for the [0.1.0 product +contract](../docs/release-0.1.0.md). The other design documents define local +APIs and algorithms. This document defines the work order and the proof that +the parts work together. + +## Design rules + +The main product is one process with one local store. + +All input and output uses an interface. All time comes from the injected +`Clock`. A channel, not a mutex, waits for another call. Each component is an +explicit state machine. These rules make failure tests repeatable. + +The application owns its local business records. The Runtime owns Grain State +and Reminder records. The Runtime does not combine application records with +these runtime records. + +Future cluster work needs clear GrainId, Grain Reference, Call, State, and +encoding boundaries. Cluster health and cluster operations are not 0.1.0 +gates. + +## Work packages + +### 1. Freeze the contract + +Write one table for these rules: + +- GrainId, GrainType, and GrainKey names; +- Call order, overload, timeout, cancel, panic, and the non-reentrant rule; +- start and deactivation transitions; +- State read, write, Exists, Clear, conflict, unknown store result, and restart; +- Reminder setting, claim, cancel, tick status, and method failure; +- background error sources and call observations; +- application data and safe repeat rules. + +The table must link to the detailed documents. It must not copy API text. +Behavior not in the table is not a release promise. + +### 2. Harden the Grain runtime + +Use one call path for local Calls and future remote Calls. The root Runtime +must decide if it still accepts a Call before the Call reaches a Grain. + +For each GrainId, start, queue, method run, and leave must follow one state +machine. + +Tests must cover: + +- two callers that start one key at the same time; +- a method panic with calls in the queue; +- cancel while a call waits and while a method runs; +- start failure and state write failure; +- normal close and forced stop; +- leave reason and background error reporting; +- a type name that does not depend on an incidental Go type string. + +The runtime does not retry a business method. The caller must decide if a +retry is safe. + +### 3. Harden state and Reminders + +Keep State and Reminders as separate interfaces. Keep the version check on +State writes. A Reminder claim must select one delivery attempt. `Exists` and +`Clear` must distinguish absent State from a present zero value. + +Tests and simulation must cover: + +- due work found after a process restart; +- two pollers that claim one record at the same time; +- a failed claim; +- process failure after claim and before method entry; +- a failed Reminder method; +- cancel and reset; +- a periodic Reminder after downtime without replaying every missed time. +- State that is absent, present with a zero value, and cleared; +- first tick time, period, and current tick time for a periodic Reminder. + +The result is at-most-once delivery. Recovery is an application pattern: +save a pending action, wake a fixed Grain, and make the handler safe to run +more than once. The runtime gives the Reminder and call paths. It does not own +the business record. + +### 4. Complete call data and encoding + +Request Context is data that travels with a Call, such as a trace ID. Add the +smallest call path that supports it: + +- the caller can add Request Context; +- the method can read incoming Request Context; +- local Calls and future remote Calls use the same call path; +- no shared Call Filter pipeline is part of 0.1.0; +- the call path cannot bypass call admission or change call order; +- stable error codes cross the call boundary; +- cancel rules stay clear after a method was sent. + +Keep JSON as the current encoding. Do not add an encoding plug-in or a +zero-downtime upgrade format without a real user need. Typed interfaces, +stable type names, and written compatibility rules define the application +contract. + +### 5. Build the integration sample + +Add a small example and tests that use only public APIs. It must contain: + +- a Grain with current State; +- a fixed-key coordinator or dispatcher; +- a saved pending action; +- application records in its own local rows; +- a periodic recovery Reminder; +- a handler that is safe to run more than once; +- a process stop between save and delivery; +- a repeated delivery that does not apply the business change twice. + +The example is not a second framework. It proves that the public Runtime +boundaries support a real durable application pattern with local application +data and safe repeat handling. + +### 6. Run release checks + +The release candidate must pass: + +- `make test`; +- `make sim`; +- `make gen`; +- `make net`; +- `make lint`; +- `make ci`; +- race tests; +- a clean-module example build and run. + +Storage tests that measure disk durability must use a real-disk path. The +simulation must report nonzero test cases and repeat the full seed set with +the same result. + +## Failure table + +| Failure | Runtime result | What the application must do | +| --- | --- | --- | +| Queue full | The call is rejected. The method does not start. | Apply back pressure or retry when safe. | +| Caller timeout or cancel | The caller stops waiting. The method may continue. | Use a safe repeat rule or a repair action before retry. | +| Method panic | The call fails. The Grain instance is removed. | Fix the method and decide if a retry is safe. | +| State conflict | The write fails. The old instance is not trusted. | Read again and use a clear retry rule. | +| Store result is unknown | The call fails. The write may or may not exist. | Check the stored result before a non-repeatable action. | +| Claim succeeds, then process stops | The Reminder may be missed. | Save a pending action and recover it safely. | +| Reminder method fails | The error goes to the background sink. No hidden retry starts. | Save retry state in the application when needed. | +| Normal close | New calls are rejected. Accepted calls follow close rules. | Stop new outside work and drain as needed. | +| Forced stop | Queued calls are rejected. A running method may finish later. | Recover work whose result is unknown. | + +The table must state unknown results. The runtime must not turn an unknown +result into a false success or an unsafe retry. + +## API usability gate + +The first example is the main usability test. A user must be able to: + +1. declare a typed Grain interface; +2. create named State in a factory; +3. get a Grain Reference by type and key; +4. call another Grain with the Binder; +5. add Request Context and read it in the called Grain; +6. set and clear a Reminder; +7. set an error sink and Call observation; +8. close and reopen the Runtime. + +Each step must have one clear public path. The example must not need cache +details, private store layout, or a second hidden retry loop. + +## Work order + +Work lands in reviewable batches. Stop after each batch: + +1. contract and acceptance table; +2. Grain runtime edge rules; +3. state and Reminder failure rules; +4. call data and encoding boundaries; +5. integration sample and failure tests; +6. release docs and clean-install check. + +The first batch is documentation only. Code work starts after review of the +contract. + +## Gap + +The current design documents cover most single-node parts. This document adds +one failure table, one integration sample, and proof that the parts work +together after restart, failure, and a safe repeat. diff --git a/design/release.md b/design/release.md index 8d1581f..cc9abbd 100644 --- a/design/release.md +++ b/design/release.md @@ -27,7 +27,7 @@ A milestone is the set of issues committed to one tag. It is the only planning o A **0.0.x** tag (publicly visible, not announced — see "0.0.x") may be cut when its milestone has zero open items and `make ci` is green on the candidate commit. GitHub counts open issues and open pull requests against a milestone; both must be closed. No date, no issue count, no feature threshold: the batch committed to the milestone is done. This is the default release kind while gor is pre-announcement. -An **announced release** (v0.1.0 and later) is cut by maintainer judgment, not by a checklist — see "The bar for v1.0.0" and the 0.0.x note that readiness is complete and announcing is a choice. The 0.0.x chain leads up to it; the maintainer decides when to stop tagging 0.0.x and announce. +An **announced release** (v0.1.0 and later) is cut after its product contract and release-specific evidence gates are complete. For v0.1.0, those gates are [docs/release-0.1.0.md](../docs/release-0.1.0.md) and [release-0.1.0.md](release-0.1.0.md). The maintainer still decides when to create the tag and publish the release after the gates pass. Closing a milestone freezes its scope as a completed set; it is not the same act as pushing the tag, which stays the manual step in "Release sequence." The two may be separated — v0.0.1's milestone was closed as a scope marker, and its tag was pushed afterward. @@ -39,7 +39,7 @@ The current 0.0.x is the lowest-numbered open milestone. Route a new issue by th 2. **Is it cut-blocking?** A defect a user hits with ordinary use is cut-blocking: route it to the current 0.0.x. A written-but-unimplemented spec, or a defect that needs unusual input to trigger, is not cut-blocking: route it to the next 0.0.x (create one if none exists). Once in a milestone, an issue stays there until it is done or is moved at cut time. 3. **Otherwise it is backlog.** An issue with no milestone is recorded work not committed to any tag — future direction, environment debt with no user impact, or a gap explicitly deferred. No milestone is a legitimate, intended state, not a mistake and not a queue to drain before a release. -The one fork not decidable from the issue alone is "ordinary use" versus "unusual input" in question 2. That is a judgment about how likely a user is to hit the defect. Worked example: a codegen import collision triggered by two method-signature packages sharing a name (common — cut-blocking, current 0.0.x) versus one triggered only by an entity package literally named `context` (rare — next 0.0.x). The agent proposes a placement; the maintainer confirms that single fork. Seeding a milestone's initial batch, deciding to announce, and judging an issue misrouted when its work cannot wait are also maintainer calls; routing, straggler-moving, and the zero-open cut check are mechanical. +The one fork not decidable from the issue alone is "ordinary use" versus "unusual input" in question 2. That is a judgment about how likely a user is to hit the defect. Worked example: a codegen import collision triggered by two method-signature packages sharing a name (common — cut-blocking, current 0.0.x) versus one triggered only by a Grain package literally named `context` (rare — next 0.0.x). The agent proposes a placement; the maintainer confirms that single fork. Seeding a milestone's initial batch, deciding to announce, and judging an issue misrouted when its work cannot wait are also maintainer calls; routing, straggler-moving, and the zero-open cut check are mechanical. ### Merge order on the linear trunk @@ -87,7 +87,7 @@ gor is in the 0.0.x band. A 0.0.x tag is publicly visible — the repository is The 0.0.x release-note question has a zero-maintenance answer: none is written. The `release-note` blocks in merged PRs still accumulate as raw material; their only consumer is the maintainer writing the first announced release's note. Nothing is assembled, published, or kept in sync per 0.0.x tag. -The readiness work for an announced release is already complete. Every [ROADMAP.md](../ROADMAP.md) "required" item is done — English documentation, public API doc comments, the error and cancellation contract, the root runtime shutdown contract, deactivation reasons and the background error sink, the example application, observability, and the performance baseline — and `make ci` passes. The benchmark failure that had blocked the baseline is fixed (`cluster.New` honors the six probe-parameter defaults from [design/cluster.md](cluster.md), `make bench` passes on a real-disk path, the forwarding baseline re-verified on 2026-08-06), and the cluster startup snippet in [docs/programming-model.md](../docs/programming-model.md) runs verbatim in a clean module. The reason gor is at 0.0.x and not announced is the maintainer's judgment that it is not time, not a missing technical gate. Inventing a new checklist to "earn" an announced release would be dishonest; when the maintainer decides to announce, that decision is the gate. +The pre-announcement readiness work is complete. Every [ROADMAP.md](../ROADMAP.md) checklist item is done — English documentation, public API doc comments, the error and cancellation contract, the root runtime shutdown contract, deactivation reasons and the background error sink, the example application, observability, and the performance baseline — and `make ci` passes. The 0.1.0 contract adds a release-specific composition gate: the subsystem promises must be verified together under restart, failure, and duplicate delivery, and a public-API conformance application must demonstrate the integration boundary. Those requirements are not retroactively claimed by the old checklist. Multi-node is still a preview capability and partitions can misjudge healthy nodes, so no release — 0.0.x or announced — should be treated as settled. @@ -118,4 +118,4 @@ Steps 1, 2, 4, and 5 need maintainer judgment and stay manual. Step 3 already ha ## Gap -0.0.x tags are cut under the Planning rules: each is annotated, and no GitHub Release is created for any of them. The readiness work for an announced release is complete; staying in 0.0.x rather than announcing is a choice, not a missing gate. +0.0.x tags are cut under the Planning rules: each is annotated, and no GitHub Release is created for any of them. The pre-announcement checklist is complete; the additional v0.1.0 contract and evidence gates determine when the first announced release is ready. diff --git a/design/runtime.md b/design/runtime.md index 57e0256..c6e96eb 100644 --- a/design/runtime.md +++ b/design/runtime.md @@ -2,7 +2,7 @@ ## Activation -An entity's "activation" is its in-memory instance on some node. Lifecycle: +A Grain's Activation is its in-memory instance on some Silo. Lifecycle: ``` absent ── call arrives ──▶ activating ──▶ active ── idle timeout ──▶ deactivating ──▶ absent @@ -11,11 +11,11 @@ absent ── call arrives ──▶ activating ──▶ active ── idle tim reads state from the store persists ``` -Key point: **users never explicitly create or destroy entities.** `Ref[T](rt, key)` only constructs a reference; it triggers no I/O. Only the first method call triggers activation. +Key point: **users never explicitly create or destroy Grains.** `Ref[T](rt, key)` only constructs a reference; it triggers no I/O. Only the first method call triggers activation. ## Lifecycle hooks -Two optional interfaces; the runtime calls whichever the entity implements: +Two optional interfaces; the runtime calls whichever the Grain implements: ```go type Activatable interface { @@ -36,26 +36,29 @@ const ( ) ``` -**Use optional interfaces, not required methods.** Most entities need neither; making them write two empty methods would be pure ceremony. Passing functions at registration time is out too — it would put "what this entity does on activation" a mile away from the entity itself. +**Use optional interfaces, not required methods.** Most Grains need neither; making them write two empty methods would be pure ceremony. Passing functions at registration time is out too — it would put "what this Grain does on activation" a mile away from the Grain itself. -`OnActivate` runs after state is read back from the store and before the first call enters the mailbox. If it returns an error, activation failed: the call that triggered this activation gets the error, the activation is not established, the placeholder closes with the error, and the next call starts over. There is no "half-activated" intermediate state — an entity that failed `OnActivate` yet still serves is worse than having no hook at all. +`OnActivate` runs after state is read back from the store and before the first call enters the mailbox. If it returns an error, activation failed: the call that triggered this activation gets the error, the activation is not established, the placeholder closes with the error, and the next call starts over. There is no "half-activated" intermediate state — a Grain that failed `OnActivate` yet still serves is worse than having no hook at all. `OnDeactivate` runs right before the instance disappears, after the mailbox has been drained. It receives the `DeactivationReason` that first started the deactivation. There are only four reasons: | Reason | What first triggers deactivation | What the app can do with it | | --- | --- | --- | | `Idle` | The instance idles past the timeout. | Don't treat a local reclamation as the business object going offline. | -| `OwnershipLost` | The current node no longer owns the identity, or the view has no active owner. | Release node-local leases or connections; don't announce that the business object is gone. | +| `OwnershipLost` | The current node no longer owns the GrainId, or the view has no active owner. | Release node-local leases or connections; don't announce that the business object is gone. | | `RuntimeClosed` | The root runtime begins a graceful stop. | Do teardown before the process exits. | -| `Faulted` | A method panicked, or the entity asked to discard the current instance. | Don't treat an untrusted instance as a normal farewell; raise the alert level. | +| `Faulted` | A method panicked, or the Grain asked to discard the current instance. | Don't treat an untrusted instance as a normal farewell; raise the alert level. | This is the complete public set. A value may join the set only if it forces the app to make a different decision; a reason must not be added just because the implementation gained a branch. Panic and discard both mean the current instance is no longer trustworthy; migration and no-owner both mean the current node loses ownership — hence one value each. The reason is written in the same atomic transition where `beginDeactivation(reason)` moves the activation from `active` to `deactivating`. Later events must not overwrite it once the activation is already deactivating. If the root runtime has entered `closing`, one activation may have already started deactivating for `Idle`; its reason stays `Idle`. A deactivation reason describes why an activation first leaves; the root state machine describes whether the whole runtime admits calls, how it waits, and with which stop error it rejects calls. These are two concepts and must not share one enum. -**Returning an error changes nothing.** Deactivation cannot be rejected, and the state is in the store anyway. The error has no caller; like scheduled delivery failures, it goes to the runtime's error sink (see [timers.md](timers.md)), with no retry. The sink's source carries `Deactivation{Reason: reason}` instead of a fabricated method name. +**Returning an error changes nothing.** Deactivation cannot be rejected, and +State is in the store anyway. The error has no caller. Like Reminder delivery +failures, it goes to the Runtime error sink (see [timers.md](timers.md)), with +no retry. The sink source carries `Deactivation{Reason: reason}`. -Each normal deactivation hook gets a fresh `context.Background()`. This context has no deadline and is never canceled; it inherits nothing from any caller of the entity. A graceful stop waits for hooks that already started, so a hook must finish promptly. +Each normal deactivation hook gets a fresh `context.Background()`. This context has no deadline and is never canceled; it inherits nothing from any caller of the Grain. A graceful stop waits for hooks that already started, so a hook must finish promptly. **Neither `Kill()` nor this node being declared dead starts `OnDeactivate`.** An abrupt stop gives no teardown chance to hooks that have not started. Hooks that already started are not canceled, and an abrupt stop does not wait for them; handing them a canceled context would only create a third semantics of partial teardown. @@ -71,7 +74,7 @@ Each node keeps a table: ```go type activation struct { - id Identity + id GrainId instance any mailbox *mail.Box lastUsed time.Time @@ -110,11 +113,11 @@ call ──▶ ring: who owns this id? ── self ──▶ runtime ──▶ l transport.Send ──▶ remote node ``` -**The fork is in `gor`, not in `runtime`.** `runtime` only sees the left branch: give it an Identity, it finds or builds the activation and delivers the call into the mailbox. It does not know the right branch exists — and so single-node mode has no extra code to route around (see [cluster.md](cluster.md)). +**The fork is in `gor`, not in `runtime`.** `runtime` only sees the left branch: give it a GrainId, it finds or builds the activation and delivers the call into the mailbox. It does not know the right branch exists — and so single-node mode has no extra code to route around (see [cluster.md](cluster.md)). ## Reentrancy -By default, an entity does not accept a second call while processing one. +By default, a Grain does not accept a second call while processing one. This brings the classic deadlock — A calls B, B calls A back. Orleans relaxes the restriction with `[Reentrant]` / `[AlwaysInterleave]` annotations, at the cost of the user having to reason about invariants under interleaved execution. @@ -122,11 +125,11 @@ gor's stance: no reentrancy for now. A deadlock in gor shows up as a call timeou If practice proves it necessary, it will be added — at method granularity, not type granularity. -Call cycle detection requires carrying a set of already-occupied entities along the call chain. Go has no `AsyncLocal`; the only option is to carry it explicitly in `context.Context` — a genuine disadvantage of Go relative to .NET, see [research/go-capabilities.md](../research/go-capabilities.md) (in Chinese). +Call cycle detection requires carrying a set of already-occupied Grains along the call chain. Go has no `AsyncLocal`; the only option is to carry it explicitly in `context.Context` — a genuine disadvantage of Go relative to .NET, see [research/go-capabilities.md](../research/go-capabilities.md) (in Chinese). ## Errors and timeouts -Every call carries a timeout (from `ctx`). The semantics of the timeout must be stated clearly: a timeout means the caller is no longer waiting, not that the entity stops executing. The method body may already have changed state. +Every call carries a timeout (from `ctx`). The semantics of the timeout must be stated clearly: a timeout means the caller is no longer waiting, not that the Grain stops executing. The method body may already have changed state. No automatic retry is provided. The runtime does not know whether a method is idempotent; retrying on the user's behalf causes problems like duplicate charges. Retrying is the caller's decision. @@ -160,21 +163,27 @@ running ── Close ──▶ closing ── graceful stop done ──▶ stopp └── node declared dead ──▶ dead ── abrupt stop done ──▶ stopped ``` -There are only four transition functions: `beginClose` moves `running` to `closing`; `beginKill` moves `running` or `closing` to `killing`; `becomeDead` only moves a root runtime still in `running` to `dead`; `finishStop` moves `closing`, `killing`, or `dead` to `stopped`. There are no back edges. A repeated `Close` and a `Kill` after the runtime has already stopped do not change state; a `Kill` during `closing` is an escalation, not a no-op that waits out the pending `Close`. +There are only four transition functions: `beginClose` moves `running` to +`closing`; `beginKill` moves `running` or `closing` to `killing`; +`becomeDead` only moves a root Runtime still in `running` to `dead`; +`finishStop` moves `closing`, `killing`, or `dead` to `stopped`. There are no +back edges. A repeated `Close` and a `Kill` after the Runtime has stopped do +not change state. A `Kill` during `closing` is an escalation. ### Admission is the only boundary -The atomic transition by which `beginClose`, `beginKill`, or `becomeDead` successfully leaves `running` is the linearization point of call admission. It also closes the public stop signal. A closed signal is not proof that all resources are released; it only proves that no entity call can be admitted after this point. +The atomic transition by which `beginClose`, `beginKill`, or `becomeDead` successfully leaves `running` is the linearization point of call admission. It also closes the public stop signal. A closed signal is not proof that all resources are released; it only proves that no Grain call can be admitted after this point. -Every entity call first goes through the root-level `admit`. In the same serialized domain as the state transitions, it checks `running` and registers this call, and returns a release that must be called on completion. An `admit` either lands before the transition and becomes an admitted call, or lands after it and immediately gets the stop error. It must not read the state first and enter the execution runtime or start forwarding afterwards. +Every Grain call first goes through the root-level `admit`. In the same serialized domain as the state transitions, it checks `running` and registers this call, and returns a release that must be called on completion. An `admit` either lands before the transition and becomes an admitted call, or lands after it and immediately gets the stop error. It must not read the state first and enter the execution runtime or start forwarding afterwards. The following entry points all use the same `admit`; none has its own closing check: - The public `Runtime.Invoke` admits before ownership and forwarding. - The inbound `invoke` handler admits before handing to the local execution runtime. It must not call the inner execution runtime directly. -- Scheduled deliveries still go through the root call entry, so they are bound by the same rule. +- Reminder deliveries still go through the root Call entry, so they are bound + by the same rule. -Probes are not entity calls and do not count toward the call count; but they read the same root state and refuse to reply when it is not `running`. When an inbound request is rejected for stopping, it is not first checked against a separate `Done` check, and the result does not differ by local or forwarded origin. +Probes are not Grain calls and do not count toward the call count; but they read the same root state and refuse to reply when it is not `running`. When an inbound request is rejected for stopping, it is not first checked against a separate `Done` check, and the result does not differ by local or forwarded origin. `closing`, `killing`, and the `stopped` reached from either of them all return the root package's `ErrRuntimeClosed`, whose stable code is `gor.runtime_closed`. `dead` and the `stopped` reached from it return `gor.node_dead`. The cross-node reconstruction rules for these two errors are in `errors.md`; direct and forwarded calls judge by the same stable code. Internal mailbox, execution runtime, or transport errors must not supersede this root-level admission result. @@ -196,7 +205,7 @@ Thus the inner execution runtime can still drain while `closing`, but it is no l ### Gap -The root runtime's stop state machine is implemented: the four transition functions `beginClose`, `beginKill`, `becomeDead`, `finishStop`, with the atomic `admit`/release as the only admission gate; the public `Runtime.Invoke`, the inbound `invoke` handler, and scheduled deliveries share the same entry and admit before ownership and forwarding. `closing`/`killing` and the `stopped` reached from them return `gor.runtime_closed`; `dead` and the `stopped` reached from it return `gor.node_dead`. +The root runtime's stop state machine is implemented: the four transition functions `beginClose`, `beginKill`, `becomeDead`, `finishStop`, with the atomic `admit`/release as the only admission gate; the public `Runtime.Invoke`, the inbound `invoke` handler, and Reminder deliveries share the same entry and admit before ownership and forwarding. `closing`/`killing` and the `stopped` reached from them return `gor.runtime_closed`; `dead` and the `stopped` reached from it return `gor.node_dead`. Stop coordination is implemented as pure channel waiting: the execution runtime exposes `BeginClose`/`BeginKill` plus a `Done()` channel, the cluster node exposes a `DeclaredDead()` channel, and the root coordinator in `closeGracefully`/`closeImmediately` receives only `clusterDone`, `engine.Done()`, `drained`, and `transportDone`. The inner execution runtime supports the `closing → killing` escalation (`BeginKill` from `closing` is not a no-op: it closes the `killing` channel, marks deactivation hooks that have not started to be skipped, and cancels execution). The cluster node explicitly reports "declared dead externally" via `DeclaredDead()` rather than "exited on its own", and the root layer no longer infers the reason from whether it initiated the stop itself. A declared-dead node no longer publishes the final empty view, so graceful migration and abrupt stop do not race to start. @@ -204,6 +213,6 @@ Transport teardown meets the invariant above. `closeTransport` routes by stop mo Deactivation reasons are implemented: `activation` saves the reason in the same atomic transition of `beginDeactivation(reason)`; `waitForDeactivation` and `skipOnDeactivate` read it in the same critical section and hand it to the hook. The reason is written only in that transition; later events (including the root runtime having entered `closing`) do not overwrite it. The four entry points map one-to-one onto the table above: idle eviction passes `Idle`, `Deactivate` (view eviction or no active owner) passes `OwnershipLost`, `beginStopDeactivationsLocked` passes `RuntimeClosed`, and `stopActivation` for panic and discard passes `Faulted`. Each hook gets a fresh `context.Background()` (no deadline, never canceled, inheriting no caller context); `Kill()` and being declared dead still skip hooks that have not started, and hooks that already started are neither canceled nor waited for. Hook errors are reported through the structured sink, with source `Deactivation{Reason: reason}`; see [timers.md](timers.md). -Call cycle detection is implemented: each call carries the chain of entities it already occupies in its context; forwarded requests carry the chain on the wire; and a call whose target is already on its chain is rejected at delivery with an error naming the cycle, projected onto the stable code `gor.call_cycle`. The chain is per call, so a slow call that is not a cycle still times out as a plain timeout. +Call cycle detection is implemented: each call carries the chain of Grains it already occupies in its context; forwarded requests carry the chain on the wire; and a call whose target is already on its chain is rejected at delivery with an error naming the cycle, projected onto the stable code `gor.call_cycle`. The chain is per call, so a slow call that is not a cycle still times out as a plain timeout. The only reason `Kill()` exists is simulation tests — a real process crash does not politely call a function first. It is not a shutdown API for users; users shut down with `Close()`. diff --git a/design/scheduling.md b/design/scheduling.md index 2d69870..d4b2eb5 100644 --- a/design/scheduling.md +++ b/design/scheduling.md @@ -2,11 +2,11 @@ ## Goal -Calls on the same entity are strictly serialized; calls on different entities are fully parallel. +Calls on the same Grain are strictly serialized; calls on different Grains are fully parallel. ## Implementation -One mailbox per entity: one goroutine reads from a channel and executes in a loop. +One mailbox per Grain: one goroutine reads from a channel and executes in a loop. ```go type Box struct { @@ -31,7 +31,7 @@ func (b *Box) run() { Serialization falls out directly from the fact that only one goroutine runs the loop body. No locks, no custom scheduler. -**`reply` must have capacity 1.** After the caller times out it stops reading this channel ([runtime.md](runtime.md): a timeout means the caller is no longer waiting, not that the entity stops executing). With an unbuffered `reply`, the send would hang forever — not one call, but the whole mailbox loop; the entity would be dead from then on. Capacity 1 makes the send never block, and an unclaimed result is reclaimed together with the channel. +**`reply` must have capacity 1.** After the caller times out it stops reading this channel ([runtime.md](runtime.md): a timeout means the caller is no longer waiting, not that the Grain stops executing). With an unbuffered `reply`, the send would hang forever — not one call, but the whole mailbox loop; the Grain would be dead from then on. Capacity 1 makes the send never block, and an unclaimed result is reclaimed together with the channel. Not replaced by `select` with `ctx.Done()`: that would add a branch to every reply, and the thing it guards against is prevented by a single buffer slot. @@ -41,14 +41,14 @@ The whole `mail` package is estimated at around 100 lines. For comparison: Orlea Buffered or unbuffered directly changes the backpressure semantics: -- **Unbuffered**: the caller blocks until the entity starts processing. Backpressure propagates automatically, but one slow entity hangs all its callers. +- **Unbuffered**: the caller blocks until the Grain starts processing. Backpressure propagates automatically, but one slow Grain hangs all its callers. - **Buffered**: absorbs bursts, but behavior at a full buffer must be defined — block or reject. -The choice: bounded buffer, reject when full (returning a clear overload error). An unbounded queue disguises a memory problem as a latency problem, and blocking lets one hot entity drag down the whole process. Capacity is configurable. +The choice: bounded buffer, reject when full (returning a clear overload error). An unbounded queue disguises a memory problem as a latency problem, and blocking lets one hot Grain drag down the whole process. Capacity is configurable. ## Request order -Consecutive calls from the same caller to the same entity execute in the order they were initiated — the channel is FIFO, which holds naturally in the local case. +Consecutive calls from the same caller to the same Grain execute in the order they were initiated — the channel is FIFO, which holds naturally in the local case. **No order guarantee across nodes.** Network reordering plus reconnection breaks it, and the complexity of sequence numbers and reorder buffers is not worth it. The docs must state this explicitly; users must not be led to assume an order guarantee. @@ -61,12 +61,12 @@ That this design is fully observable by `testing/synctest` is not a coincidence So `sync.Mutex` must not appear in the `mail` package for cross-call waiting. A short critical section that merely protects a map is fine; using it to "wait" is not. -## Relationship with scheduled tasks +## Relationship with Reminders -Persisted scheduled tasks (`Schedule`) do not run on the mailbox's clock. They are "a table plus a poller": the poller finds due items and constructs an ordinary call delivered to the target entity's mailbox. +Persisted Reminders (`Reminder`) do not run on the mailbox's clock. They are "a table plus a poller": the poller finds due items and constructs an ordinary call delivered to the target Grain's mailbox. -So to the entity, a scheduled task is indistinguishable from an ordinary method call and enjoys the same serialization guarantee. +So to the Grain, a Reminder is indistinguishable from an ordinary method call and enjoys the same serialization guarantee. -**Explicitly not a repeat of Orleans Reminders v1** — an in-memory cache plus ring partitioning plus complex ownership transfer, which Orleans itself replaced with `Orleans.DurableJobs` (v2, measured at 5278 lines, still preview). Table plus polling is dumber but easier to verify, and accurate enough — persisted scheduled tasks should never promise millisecond precision. +**Explicitly not a repeat of Orleans Reminders v1** — an in-memory cache plus ring partitioning plus complex ownership transfer, which Orleans itself replaced with `Orleans.DurableJobs` (v2, measured at 5278 lines, still preview). Table plus polling is dumber but easier to verify, and accurate enough — persisted Reminders should never promise millisecond precision. Details in [timers.md](timers.md). diff --git a/design/simulation.md b/design/simulation.md index ef3f5cd..25a2767 100644 --- a/design/simulation.md +++ b/design/simulation.md @@ -30,16 +30,16 @@ A step looks like this: This is the easiest place to get wrong; it is worth stating in no uncertain terms. -Intuitively, observations at the quiescence point should be interleaving-independent: all calls are done, `Add` is commutative, and the entity's value is the same regardless of order. +Intuitively, observations at the quiescence point should be interleaving-independent: all calls are done, `Add` is commutative, and the Grain's value is the same regardless of order. -**Once faults can deactivate activations, this stops holding.** Two concurrent calls on the same entity with "the write took effect but errored" injected: the first call writes, errors, and the activation is deactivated. The second call either is still queued and gets rejected along with it (the entity's value was incremented once), or re-activates in time, reads the new value, and writes again (incremented twice). Which one happens depends on scheduling. +**Once faults can deactivate activations, this stops holding.** Two concurrent calls on the same Grain with "the write took effect but errored" injected: the first call writes, errors, and the activation is deactivated. The second call either is still queued and gets rejected along with it (the Grain's value was incremented once), or re-activates in time, reads the new value, and writes again (incremented twice). Which one happens depends on scheduling. The multiset of outcomes is the same: both `{write error, write error}` and `{write error, closed}` are possible. So: **with fault injection, no observation is interleaving-independent.** The log therefore splits in two: -- **Decisions** — what the PRNG produced: which entity this step hits, how much to add, which fault to inject, whether to crash. Fully decided by the seed. -- **Observations** — outcomes, entity values, intermediate quantities of invariant checks. For humans; when something fails, reconstruct the incident from them. +- **Decisions** — what the PRNG produced: which Grain this step hits, how much to add, which fault to inject, whether to crash. Fully decided by the seed. +- **Observations** — outcomes, Grain values, intermediate quantities of invariant checks. For humans; when something fails, reconstruct the incident from them. **The reproduction test compares only the decision half.** The observation half is still written to the log, but does not take part in the comparison. @@ -63,18 +63,22 @@ That is the defect, and it is **one root, not two**. "May a fault be consumed no ### The remedy: bind the target by the seed -A fault is two facts — a kind and a target. The kind has always been a decision. The target must be a decision too. The store fault already does this: keyed by entity identity, drawn in the driver, read fresh on every call to that entity, not consumed by first arrival. The member fault must meet the same bar. This is not new machinery; it is removing the inconsistency that left the member fault the odd one out. +A fault is two facts — a kind and a target. The kind has always been a decision. The target must be a decision too. The store fault already does this: keyed by GrainId, drawn in the driver, read fresh on every call to that Grain, not consumed by first arrival. The member fault must meet the same bar. This is not new machinery; it is removing the inconsistency that left the member fault the odd one out. -The target of a member fault is a member row — the `(node address, generation)` the member store keys on — for the write and delay kinds, and a node for the list-error kind. The driver draws the target with the seed, the same way it draws node indices for calls and crashes, resolving a node to its current generation. The fault fires only on an operation addressing that target; if none does this step it does not fire — dropped, deterministically, the way a store fault on an entity nobody calls does not manifest. The delay kind is already shape-bound to an active-refresh write; it takes the target row as well, so two survivors heartbeating no longer race for one delay token. With the target fixed by the seed, restart-success is fixed by the seed (a write or list fault fails restart exactly when it targets the restarting node; a delay never does), liveness is fixed by the seed, and the decision half is pure again. +The target of a member fault is a member row — the `(node address, generation)` the member store keys on — for the write and delay kinds, and a node for the list-error kind. The driver draws the target with the seed, the same way it draws node indices for calls and crashes, resolving a node to its current generation. The fault fires only on an operation addressing that target; if none does this step it does not fire — dropped, deterministically, the way a store fault on a Grain nobody calls does not manifest. The delay kind is already shape-bound to an active-refresh write; it takes the target row as well, so two survivors heartbeating no longer race for one delay token. With the target fixed by the seed, restart-success is fixed by the seed (a write or list fault fails restart exactly when it targets the restarting node; a delay never does), liveness is fixed by the seed, and the decision half is pure again. ### Per seam Whether a seam carries this defect turns on one test: does its first-arrival consumption move a runtime quantity the decision encoding reads? -- **Store read/write fault** — keyed by entity identity, not consumed by first arrival. No defect. -- **Schedule claim fault** — keyed by entity identity; the consuming `Claim` is CAS-unique, so the fault rides the one winner. Which node wins is scheduling, but the observable — one delivery, the fault applied — is invariant, and liveness is untouched. No defect; the residual scheduling dependence is the accepted outcome kind. +- **Store read/write fault** — keyed by GrainId, not consumed by first arrival. No defect. +- **Reminder claim fault** — keyed by GrainId; the consuming `Claim` is + CAS-unique, so the fault rides the one winner. Which Silo wins is a + scheduling detail. The observable is one delivery with the fault applied. - **Member fault** — single unkeyed field, first-arrival. Its consumption fixes restart-success, which moves liveness, which the decision encoding reads. **The defect.** -- **Schedule list fault** — single unkeyed field, first-arrival, same *shape* as the member fault. But a list error is read-only and the poller retries next tick; it moves no quantity the decision encoding reads, so no divergence reaches the decision half. **Benign today; take the same target binding for consistency, not urgency** — a future seam that let schedule state feed a decision would reopen the leak through the same shape. +- **Reminder list fault** — a single unkeyed field with first-arrival behavior. + A list error is read-only and the poller retries on the next tick. It does + not change the decision sequence today. - **Network fault** — not this shape. A partition is a deterministic group map applied per node pair (a whole pair goes silent); a per-message drop is drawn by the seed in the driver; delay is drawn unconditionally in the driver and released by the clock. None latches onto a target by first arrival. ### What does not change @@ -117,19 +121,19 @@ A node = one `runtime.Runtime`. Several nodes share one `Store`. Before [step 6] - **Crash** — drop all in-memory state, keep the store. - **Restart** — build a new `Runtime` on the same store. -**Double activation becomes testable here.** Two Runtimes sharing one store activating the same identity is double activation by itself — no network partition needed to produce it. The core risk of cluster instability is already covered by assertions at step 4; step 6 only changes the way it is produced. +**Double activation becomes testable here.** Two Runtimes sharing one store activating the same GrainId is double activation by itself — no network partition needed to produce it. The core risk of cluster instability is already covered by assertions at step 4; step 6 only changes the way it is produced. ## A crash is not Close `Close()` drains the mailbox and waits for in-flight calls to finish. That is a graceful stop. -A crash must make in-flight calls return with an error immediately, giving entities no teardown chance. So `runtime` needs one more stop path: `Kill()` — cancel all in-flight calls' contexts, close the mailbox, do not wait for draining. +A crash must make in-flight calls return with an error immediately, giving Grains no teardown chance. So `runtime` needs one more stop path: `Kill()` — cancel all in-flight calls' contexts, close the mailbox, do not wait for draining. **`Kill()` must make every goroutine exit.** This is not cleanliness: synctest panics with a deadlock report when every goroutine in the bubble blocks forever. A leaking crashed node does not leak quietly; it takes down the whole simulation test. -Go cannot kill a call that is executing a user method. `Kill()` can only cancel the context; a user method that ignores its context keeps running to completion. This differs from a real process crash, but there is no other way, and the entities in simulation tests are written by us. +Go cannot kill a call that is executing a user method. `Kill()` can only cancel the context; a user method that ignores its context keeps running to completion. This differs from a real process crash, but there is no other way, and the Grains in simulation tests are written by us. -**After a crash, wait for the fake store to finish its in-flight work.** `Kill()` cancels the context and the caller returns with the cancellation error right away, but the entity method is still asleep inside the fake store. The bubble's fake clock only advances while every goroutine is durably blocked, and once the root exits it stops outright: the sleeping goroutine can never wake, and synctest reports a leak. +**After a crash, wait for the fake store to finish its in-flight work.** `Kill()` cancels the context and the caller returns with the cancellation error right away, but the Grain method is still asleep inside the fake store. The bubble's fake clock only advances while every goroutine is durably blocked, and once the root exits it stops outright: the sleeping goroutine can never wake, and synctest reports a leak. So the fake store must be able to report "no work in hand", and the driver waits for it before every observation step. The waiting is done with a channel: once the driver blocks, the fake clock advances, and the sleep ends on its own. @@ -190,18 +194,18 @@ No, for two reasons: ## Invariants -Checked after every step. The first two are read directly from the fake store's bookkeeping, independent of entity type: +Checked after every step. The first two are read directly from the fake store's bookkeeping, independent of Grain type: - **ETags are monotonic.** A record's ETag only grows. - **Nothing is invented.** The content of storage at any moment must equal the bytes some `Write` committed. No write can be silently rewritten into something else. The third needs a porcupine model: -- **The call history of a single entity is linearizable.** +- **The call history of a single Grain is linearizable.** ## How porcupine plugs in -The entity under test is a counter: `Add(ctx, n) (int64, error)`, returning the value after the add. The sequential spec fits in three lines, while the interleavings are many — just right. +The Grain under test is a counter: `Add(ctx, n) (int64, error)`, returning the value after the add. The sequential spec fits in three lines, while the interleavings are many — just right. Each operation in the history records call time, return time, input, and output; times come from the bubble's fake clock. @@ -213,7 +217,7 @@ One event per line, plain text, human-readable. Decisions and observations have ``` seed=8f3c2a1b -0000 decision entity=Counter/a deltas=[3,5] fault=write.applied-then-error +0000 decision Grain=Counter/a deltas=[3,5] fault=write.applied-then-error observe outcomes=[store-write-applied-then-error,closed] observe state Counter/a=3 0001 decision crash node=1 @@ -255,9 +259,9 @@ One wrinkle, faced honestly. Replay divergence is itself scheduling-dependent, s ## What the later steps hang on the skeleton -Step 5: the scheduled-task table is a new fault source (scan failures, claim failures, the claim landed but the reply was lost), with a new invariant: "one delivery per due time". See [timers.md](timers.md). +Step 5: the Reminder table is a new fault source (scan failures, claim failures, the claim landed but the reply was lost), with a new invariant: "one delivery per due time". See [timers.md](timers.md). -Step 6a: the membership table is yet another fault source, shaped like the scheduled-task table. New invariants: +Step 6a: the membership table is yet another fault source, shaped like the Reminder table. New invariants: - **Membership views eventually converge.** After faults stop, all live nodes compute the same view. - **After convergence, one key belongs to one node.** During convergence there may be more than one: that is the acknowledged double-activation window, not a bug. @@ -265,7 +269,7 @@ Step 6a: the membership table is yet another fault source, shaped like the sched "Live nodes" means nodes that **still consider themselves alive**, not nodes the driver did not crash. A persistently slow membership table makes nodes declare each other dead until everyone self-terminates (see [cluster.md](cluster.md)); that is 6a's known failure mode, and with no owner left at all it must not count as a broken invariant. After everyone is dead, the restart action brings nodes back: a fresh generation, a new row, and convergence must still happen. -**Owner uniqueness must be checked with a batch of probe identities**, not just the two under test. `Owns` is pure computation; it writes nothing to storage and activates nothing, so a few dozen keys cost nothing, and checking a full batch equals comparing views: whenever two nodes' views differ, some key's owner necessarily disagrees. This also avoids opening a "hand over the view" method on the runtime for tests. +**Owner uniqueness must be checked with a batch of probe GrainIds**, not just the two under test. `Owns` is pure computation; it writes nothing to storage and activates nothing, so a few dozen keys cost nothing, and checking a full batch equals comparing views: whenever two nodes' views differ, some key's owner necessarily disagrees. This also avoids opening a "hand over the view" method on the runtime for tests. **After ownership filtering, `claim-lost` is no longer an event every seed batch hits.** A non-owner poller never claims; two pollers claiming the same row is only possible inside the inconsistent-view window. This is the result of [timers.md](timers.md)'s rule, not a coverage regression. diff --git a/design/testing.md b/design/testing.md index 3f1a99c..cfd7413 100644 --- a/design/testing.md +++ b/design/testing.md @@ -66,8 +66,8 @@ Dropping ticks imposes a hard requirement: **clock subscription must happen in t **Invariant assertions** — checked after every step. The core ones: - State is never silently overwritten (a double activation hitting the ETag must report a conflict). -- The call history of the same entity is linearizable (handed to porcupine). -- Scheduled tasks are not delivered twice. +- The call history of the same Grain is linearizable (handed to porcupine). +- Reminders are not delivered twice. - Membership views eventually converge. **Seed reproduction** — on failure, print the seed; re-running with the same seed must produce a byte-identical event sequence. This rule itself needs a test. diff --git a/design/timers.md b/design/timers.md index 143fac0..9835461 100644 --- a/design/timers.md +++ b/design/timers.md @@ -1,70 +1,89 @@ -# Scheduled tasks +# Reminders -One table plus one poller. The poller finds rows that have come due and delivers an ordinary call to the target entity. +One table plus one poller. The poller finds rows that have come due and delivers an ordinary call to the target Grain. -The entity sees no difference: a due call goes through the same mailbox as any other call, equally serialized. +The Grain sees no difference: a due call goes through the same mailbox as any other call, equally serialized. ## The method name, not a function value ```go type account struct { balance gor.State[int64] - schedule gor.Schedule[Account] + reminder gor.Reminder[Account] } func newAccount(b *gor.Binder) *account { return &account{ balance: gor.NewState[int64](b, "balance"), - schedule: gor.NewSchedule[Account](b), + reminder: gor.NewReminder[Account](b), } } func (a *account) Open(ctx context.Context) error { - return a.schedule.Set(ctx, "monthly-interest", gor.Every(30*24*time.Hour), gor.Handle(Account.ApplyInterest)) + return a.reminder.Set(ctx, "monthly-interest", gor.Every(30*24*time.Hour), gor.Handle(Account.ApplyInterest)) } -func (a *account) ApplyInterest(ctx context.Context) error { ... } +func (a *account) ApplyInterest(ctx context.Context, tick gor.TickStatus) error { ... } ``` -**Only names can be stored in the table.** After a process crash, nobody can deserialize a closure back; so the scheduled task records the method name, and the poller sends an ordinary call by that name. What lives in the table is a string; nothing below changes that. +**Only names can be stored in the table.** After a process crash, nobody can deserialize a closure back; so the Reminder records the method name, and the poller sends an ordinary call by that name. What lives in the table is a string; nothing below changes that. -**But authoring is typed, not a string.** The last argument to `Set` is a method handle built from a Go method expression on the entity's interface: +**But authoring is typed, not a string.** The last argument to `Set` is a method handle built from a Go method expression on the Grain's interface: ```go -type Schedule[T any] struct { /* bound to one entity identity */ } +type Reminder[T any] struct { /* bound to one GrainId */ } + +type TickStatus struct { + FirstTickTime time.Time + Period time.Duration + CurrentTickTime time.Time +} type MethodHandle[T any] struct { /* unexported: the method name */ } -func Handle[T any](m func(T, context.Context) error) MethodHandle[T] +func Handle[T any](m func(T, context.Context, TickStatus) error) MethodHandle[T] -func (s Schedule[T]) Set(ctx context.Context, name string, when ScheduleTime, m MethodHandle[T]) error +func (s Reminder[T]) Set(ctx context.Context, name string, when ReminderTime, m MethodHandle[T]) error ``` -`Account.ApplyInterest` is a Go method expression. The compiler checks that `ApplyInterest` is a method of `Account` and that its signature is `func(Account, context.Context) error`; a typo, a rename, or a signature drift is a compile error at the call site — not a failure hours later at delivery. The type parameter ties the handle to the schedule's entity: a handle built from another entity's interface does not assign to `MethodHandle[Account]`, so it cannot reach an `Account`'s schedule. +`Account.ApplyInterest` is a Go method expression. The compiler checks that +`ApplyInterest` is a method of `Account` and that its signature is +`func(Account, context.Context, TickStatus) error`. A typo, rename, or +signature drift is a compile error. The type parameter ties the handle to the +Grain's interface. -The name the table stores is read off the method expression once, when `Handle` is called, with `reflect` and `runtime.FuncForPC`. The format of that name is not a Go-documented contract — it is empirically stable across the Go versions in use, but Go is free to change it. The implementation must therefore carry a unit test that locks the map from an interface method expression to its trailing-segment method name, so a Go upgrade that changes the encoding breaks the test instead of silently mis-naming schedules. That runs at scheduling setup, never on the delivery path; what the poller reads at delivery is the same `method` column as before. `Handle` takes a method expression on the entity interface — a hand-written closure of the same function type also compiles, but the name read off it is not a real method name and delivery fails with "unknown method". The contract is stated, not guarded: the type signature admits only `func(T, context.Context) error`, and the rest is the caller using the documented form. +The name the table stores is read off the method expression once, when `Handle` is called, with `reflect` and `runtime.FuncForPC`. The format of that name is not a Go-documented contract — it is empirically stable across the Go versions in use, but Go is free to change it. The implementation must therefore carry a unit test that locks the map from an interface method expression to its trailing-segment method name, so a Go upgrade that changes the encoding breaks the test instead of silently mis-naming Reminders. That runs at scheduling setup, never on the delivery path; what the poller reads at delivery is the same `method` column as before. `Handle` takes a method expression on the Grain interface — a hand-written closure of the same function type also compiles, but the name read off it is not a real method name and delivery fails with "unknown method". The contract is stated, not guarded: the type signature admits only `func(T, context.Context, TickStatus) error`, and the rest is the caller using the documented form. -The called method must be in the entity's interface — the dispatch table is produced by the generator from the interface, and a method not in it cannot be found at delivery time. Every interface method already has a dispatch case, so a method expression that compiles is a method delivery can find. +The called method must be in the Grain's interface. The generated dispatch +table must accept the `TickStatus` value at delivery time. -**Why a method expression, not a generated handle value.** Schedules are set from inside entity methods, which live in the entity package. The entity package cannot import the package generated from its own interfaces: that package imports the entity package for the interface types used in its proxies and dispatch, so the import is a cycle — the same reason generated artifacts land in their own package that the entity package does not import (see [codegen.md](codegen.md)). A per-method handle symbol emitted by the generator therefore cannot be named from the code that sets a schedule. The method expression is the only compile-time-checked way to name a method from that code using just the `gor` package and the interface declared in the entity package, so the handle carries no generated symbol. The generator changes nothing for this. +**Why a method expression, not a generated handle value.** Reminders are set from inside Grain methods, which live in the Grain package. The Grain package cannot import the package generated from its own interfaces: that package imports the Grain package for the interface types used in its proxies and dispatch, so the import is a cycle — the same reason generated artifacts land in their own package that the Grain package does not import (see [codegen.md](codegen.md)). A per-method handle symbol emitted by the generator therefore cannot be named from the code that sets a reminder. The method expression is the only compile-time-checked way to name a method from that code using just the `gor` package and the interface declared in the Grain package, so the handle carries no generated symbol. The generator changes nothing for this. -"One unified entry point" is rejected. Orleans has entities implement `ReceiveReminder(name)` and switch on the name themselves — that is bringing back the hand-written dispatcher deleted at [step 3](../ROADMAP.md#3-typed-proxy-code-generation), and in user code of all places. gor's selling point is compile-time typing; it must not open a string-dispatch loophole here. +"One unified entry point" is rejected. Orleans has Grains implement `ReceiveReminder(name)` and switch on the name themselves — that is bringing back the hand-written dispatcher deleted at [step 3](../ROADMAP.md#3-typed-proxy-code-generation), and in user code of all places. gor's selling point is compile-time typing; it must not open a string-dispatch loophole here. ## The stored identifier is the method name The `method` column holds the method name read off the handle — exactly the string the old string API held. A typed handle changes how the name is authored, not what is stored; the table, the poller, and cross-restart recovery do not change. -A method rename invalidates already-scheduled rows: the stored name no longer matches a dispatch case, and delivery returns "unknown method". gor does not introduce a separate stable id for the user to maintain alongside the method — that is a mapping the user must keep correct, and the library keeps its model to the necessary properties. The identifier follows the method name; a rename is a breaking change to scheduled tasks, stated plainly rather than papered over. +A method rename invalidates existing Reminder rows: the stored name no longer +matches a dispatch case, and delivery returns "unknown method". The +identifier follows the method name. A rename is a breaking change to +Reminders. -The hazard is bounded. Where an entity re-asserts its schedules on activation or first use, the rename self-heals on the next activation: `Set` overwrites the row by name, including its `method`, so the new name replaces the old. A one-shot task waiting in the table across a rename is the real casualty — it fails once at delivery, the error reaches the configured sink, and the user sets it again. +The hazard is bounded. Where a Grain re-asserts its Reminders on activation or first use, the rename self-heals on the next activation: `Set` overwrites the row by name, including its `method`, so the new name replaces the old. A one-shot Reminder waiting in the table across a rename is the real casualty — it fails once at delivery, the error reaches the configured sink, and the user sets it again. ### Migration -This is a planned v0 breaking change. `Schedule` becomes `Schedule[T]`; `NewSchedule(b)` becomes `NewSchedule[Account](b)`. Each `Set` call site changes its last argument from a string literal to `gor.Handle(InterfaceName.MethodName)`; `Cancel` is unchanged, since it takes the task name. The `method` column and the table are unchanged, so already-scheduled rows survive a restart across the upgrade — only source migrates, no data does. There is no deprecation period: at 0.0.x the call surface may change, and a typed handle and a string name sharing one parameter would only postpone the same edit. +This is a planned v0 breaking change. The public type is `Reminder[T]`, and +the constructor is `NewReminder[T]`. Each `Set` call uses +`gor.Handle(InterfaceName.MethodName)`. `Cancel` takes the Reminder name. The +stored method name and table shape do not change, so existing rows survive a +restart. Only source code needs migration. ## The handle comes from the Binder -Like `State`: `gor.NewSchedule[Account](b)` binds identity and storage at entity construction, and the methods use it directly afterwards. +Like `State`, `gor.NewReminder[Account](b)` binds the GrainId and storage at +Grain construction. The methods use it directly afterwards. Not fished out of `ctx`. Hiding runtime capabilities in `context.Value` makes "what this code needs" invisible and forces tests to build the right ctx before they can run. Constructor parameters are explicit; ctx is not. @@ -77,21 +96,24 @@ gor.After(d) // one-shot, fires once after d gor.Every(d) // periodic, fires every d ``` -`Set` overwrites by name: only one task per name on the same entity; setting again reschedules rather than adding another. `Cancel(ctx, name)` deletes it. +`Set` overwrites by name. One Grain has only one Reminder with a given name. +Setting it again changes its time. `Cancel(ctx, name)` deletes it. **Missed windows are not made up.** If the process is down for three periods, it fires once on return and then tracks to the next future time. Making up three firings is a trap — what users want is almost never "run everything that piled up", and how much piles up depends on the downtime, making the behavior unpredictable. If catch-up is truly wanted, users compute it in the method from the last execution time. -**Precision is the polling interval.** Persisted scheduled tasks should never promise milliseconds. +**Precision is the polling interval.** Persisted Reminders should never promise milliseconds. ## The table ``` -schedule(entity_type, entity_key, name, method, due_at, interval, etag) +reminder(grain_type, grain_key, name, method, due_at, interval, etag) ``` A zero `interval` means one-shot. -The primary key is (entity_type, entity_key, name). `name` identifies the task; `method` is the method to call when due — both are needed, because the same method can back several tasks with different periods. +The primary key is (GrainType, GrainKey, name). `name` identifies the +Reminder. `method` is the method to call when due. One method can back +several Reminders with different periods. ## The table's interface @@ -104,7 +126,7 @@ Four operations, aligned with what the poller and the user each need to do: **The etag exists only for claiming.** The user's `Set` / `Cancel` carries no etag: the user does not have one anyway, and an explicit reschedule or cancel is his to win. The claim that got overwritten simply delivered one fewer time; at-most-once still holds. -**The next due time is computed by the poller, not the table.** "No catch-up for missed" is policy; the table is only responsible for getting the CAS right. One-shot tasks use the zero time for "no next" — the same convention as a zero `interval`. +**The next due time is computed by the poller, not the table.** "No catch-up for missed" is policy; the table is only responsible for getting the CAS right. One-shot Reminders use the zero time for "no next" — the same convention as a zero `interval`. No row-count limit on "list due". Add it when it is actually needed; adding it now decides for a scale that does not exist yet. @@ -112,7 +134,8 @@ No row-count limit on "list due". Add it when it is actually needed; adding it n The poller scans rows with `due_at <= now` and, for each row: -1. **Claim** — CAS to push `due_at` to the next period (delete the row for one-shot tasks). +1. **Claim** — CAS to push `due_at` to the next period (delete the row for + one-shot Reminders). 2. Deliver the call only after winning the claim. **Push to the first time still in the future**, not `due_at + interval`. After three periods of downtime, adding one interval still lands in the past; the next scan hits the same row again, and "no catch-up" becomes catch-up. @@ -123,13 +146,13 @@ The reverse order causes repeated firing on a crash. Crash after the claim but b Delivery failures are not retried. Only the user knows whether retrying is safe; the runtime does not decide for him — the same stance as on `State.Set()` conflicts. -**But no retry does not mean silence.** A scheduled delivery has no caller waiting; the error returned by the method is sent by the runtime to the configured error sink, and dropped only when no sink is configured. The runtime does not retry for the user; whether to alert remains the user's decision. +**But no retry does not mean silence.** A Reminder delivery has no caller waiting; the error returned by the method is sent by the runtime to the configured error sink, and dropped only when no sink is configured. The runtime does not retry for the user; whether to alert remains the user's decision. So the configuration needs an error sink: ```go type BackgroundError struct { - Identity Identity + GrainId GrainId Err error Source ErrorSource } @@ -138,11 +161,11 @@ type ErrorSource interface { errorSource() } -type ScheduledInvocation struct { +type ReminderInvocation struct { Method string } -func (ScheduledInvocation) errorSource() {} +func (ReminderInvocation) errorSource() {} type Deactivation struct { Reason DeactivationReason @@ -153,17 +176,21 @@ func (Deactivation) errorSource() {} func OnError(func(BackgroundError)) Option ``` -`ErrorSource`'s unexported method seals the set inside the `gor` package; code outside the package cannot implement new sources. Every event is constructed by `gor`, and there are only two sources: claimed scheduled deliveries use `ScheduledInvocation{Method: ...}`, deactivation hook failures use `Deactivation{Reason: ...}`. Callers branch on the concrete type of `Source`, not on comparing writable strings; so a scheduled method exactly named `"OnDeactivate"` cannot be confused with a deactivation source. +`ErrorSource`'s unexported method seals the set inside the `gor` package. Code +outside the package cannot implement new sources. Claimed Reminder deliveries +use `ReminderInvocation{Method: ...}`. Deactivation hook failures use +`Deactivation{Reason: ...}`. Callers branch on the source type, not on a +writable string. -**One sink, not one per scheduled task, not one per event kind.** It only reports the two kinds of application callback failures with no caller to receive them: claimed scheduled deliveries and normal deactivation hooks. Direct calls return errors to the caller as usual. Polling scans, claim failures, and losing the CAS do not enter the sink — they are operational states of the scheduler and storage, not failures of a known application action. +**One sink, not one per Reminder, not one per event kind.** It only reports the two kinds of application callback failures with no caller to receive them: claimed Reminder deliveries and normal deactivation hooks. Direct calls return errors to the caller as usual. Polling scans, claim failures, and losing the CAS do not enter the sink — they are operational states of the scheduler and storage, not failures of a known application action. `Err` is exactly the error the callback got. It follows [errors.md](errors.md): across nodes, only the stable `Code` is usable with `errors.Is`; the event does not add its own `Code` field, nor does it restore error types, fields, or wrapping. -It does no retry, no backoff, no alerting policy — those are the user's business; the runtime only delivers "this failed" into the user's hands. The event carries no schedule name, due time, interval, ETag, or attempt count: after claiming these fields may already be stale, the ETag is not an application decision, and the runtime has no retry model. `timer.Invoker` keeps receiving only identity and method. +It does no retry, no backoff, no alerting policy — those are the user's business; the runtime only delivers "this failed" into the user's hands. The event carries no reminder name, due time, interval, ETag, or attempt count: after claiming these fields may already be stale, the ETag is not an application decision, and the runtime has no retry model. `timer.Invoker` keeps receiving only GrainId and method. When unconfigured, these two kinds of errors are dropped. They are the only application callback errors the runtime drops on the user's behalf, and must be written in the docs, not hidden in the implementation. -**A delivery canceled mid-shutdown is not a failure.** At runtime shutdown, in-flight scheduled calls come back with a cancellation error — the method did not fail; the runtime stopped running. Sending it to `OnError` would report a false alarm to the user on every clean shutdown, and users would have to filter cancellation errors out in their own callbacks. So when the poller's context is already canceled, this error does not go out. +**A delivery canceled mid-shutdown is not a failure.** At runtime shutdown, in-flight Reminder Calls come back with a cancellation error — the method did not fail; the runtime stopped running. Sending it to `OnError` would report a false alarm to the user on every clean shutdown, and users would have to filter cancellation errors out in their own callbacks. So when the poller's context is already canceled, this error does not go out. This is not defensive special-casing; it is behavior a test must watch: cancellations during shutdown do not enter `OnError`; other callback errors matching the sink boundary above must enter. @@ -171,25 +198,30 @@ This is not defensive special-casing; it is behavior a test must watch: cancella ### Migration -This is a planned v0 breaking change. Existing three-parameter error handlers change to receive one `BackgroundError`. Scheduled deliveries read `ScheduledInvocation.Method`; deactivation hook failures read `Deactivation.Reason`. Existing code branching on `method == "OnDeactivate"` must be deleted. +This is a planned v0 breaking change. Existing three-parameter error handlers +change to receive one `BackgroundError`. Reminder deliveries read +`ReminderInvocation.Method`; deactivation hook failures read +`Deactivation.Reason`. ### Gap -The error sink is implemented: `OnError` receives a `BackgroundError` (entity, original error, source), and the source is a sealed set — `ScheduledInvocation` carries the method name, `Deactivation` carries the deactivation reason, and the unexported method prevents additions outside the package. `Err` is the error the callback got; the event has no separate `Code` field and does not restore error types, fields, or wrapping. The cross-node stable-code contract is governed by [errors.md](errors.md); the event itself never crosses nodes. The poller does not report errors from scans, claims, or losing the CAS; when the poller's context is canceled, that delivery's cancellation error is not reported (the criterion is context state, not error shape). This section's migration is complete: the old three-parameter handlers are deleted, and production code no longer classifies errors by `method == "OnDeactivate"` (test fixtures' dispatchers dispatching deliveries by method name is delivery scheduling, not error classification). +The error sink is implemented in the current code. `OnError` receives a +`BackgroundError` with the Grain, original error, and source. The source set +is sealed. The public naming migration remains part of the 0.1.0 API work. ## Don't claim rows that are not yours -In a cluster, every node's poller scans the whole table, but a row's target entity belongs to exactly one node. Before claiming a row, ask whether it is yours; if not, skip. +In a cluster, every node's poller scans the whole table, but a row's target Grain belongs to exactly one node. Before claiming a row, ask whether it is yours; if not, skip. -Not asking loses deliveries, not duplicates them: when a non-owner wins the claim, `due_at` has already been pushed (the row is deleted for one-shot tasks), and then the call is rejected by routing — that due time is gone forever, and the owner's poller will not see it next round. +Not asking loses deliveries, not duplicates them: when a non-owner wins the claim, `due_at` has already been pushed (the row is deleted for one-shot Reminders), and then the call is rejected by routing — that due time is gone forever, and the owner's poller will not see it next round. -So the poller must ask one more thing: not just "call this method", but "do I own this identity". In a single node the answer is always yes — that is not a fake implementation; a single node indeed owns everything. +So the poller must ask one more thing: not just "call this method", but "do I own this GrainId". In a single node the answer is always yes — that is not a fake implementation; a single node indeed owns everything. With an inconsistent view, two nodes may both believe they are the owner; CAS makes one lose, and at-most-once holds. If both believe it is not theirs, this due time is deferred until the view converges; at-most-once still holds. -## Coming due activates the entity +## Coming due activates the Grain -If the target entity is not in memory, delivery activates it. This is the point of persisted scheduled tasks: without it, an evicted entity would never see its next wake-up. +If the target Grain is not in memory, delivery activates it. This is the point of persisted Reminders: without it, an evicted Grain would never see its next Reminder. ## The poller @@ -201,26 +233,31 @@ It gets its own package. `runtime` cannot host it — the poller reads tables, a ## A new I/O interface -The scheduled task table does not go through `store.Store`. That interface is "read and write one state per Identity"; scanning due rows, CAS advancement, and row deletion do not fit. +The Reminder table does not go through `store.Store`. That interface is "read and write one state per GrainId"; scanning due rows, CAS advancement, and row deletion do not fit. A new interface, shaped by what the poller actually does: list due, claim one row, write one row, delete one row. It is a new fault source on the step-4 skeleton: the fake implementation injects by seed — scan failures, claim failures, and claim succeeded but the reply was lost. The third is the most important: the claim landed but the poller does not know — exactly where duplicate delivery is most likely. -## Scheduled task writes share no transaction with state writes +## Reminder writes are separate from State writes -`schedule.Set()` writes to a different table, not the same transaction as `State.Set()`. So there is a window where the task is set but the state is not persisted. +`reminder.Set()` writes to a different table from `State.Set()`. The Reminder +may be set while the State write is not confirmed. -The two tables are not bound into one transaction so that backends are not tied together — coordination tables must be able to live on Postgres in the future, while the state table may live elsewhere. The cost is written here; users decide whether it matters. +The two tables stay separate so future backends can use different stores. The +Application must handle this partial result when both changes are needed. ## Invariants The step-4 skeleton must hold this one: -- **One delivery per due time.** For the same (entity, name) and the same `due_at`, history contains at most one delivery. +- **One delivery per due time.** For the same (Grain, name) and the same `due_at`, history contains at most one delivery. Crashes, claim failures, two pollers scanning at the same time — none of these may break it. ## Gap -The typed method handle is implemented. `Schedule` is `Schedule[T]`, `NewSchedule[T]` binds it at entity construction, and `Set`'s last parameter is a `MethodHandle[T]` built by `gor.Handle(Interface.Method)`. The method name is read off the expression once, at `Handle`, with `reflect` and `runtime.FuncForPC`; a unit test locks the map from an interface method expression to its trailing-segment method name, because `FuncForPC`'s name format is not a Go-documented contract and a Go upgrade that changes the encoding must break the test, not silently mis-name schedules. The table, the poller, and cross-restart recovery are unchanged: the `method` column still holds the method-name string, read at scheduling setup and never on the delivery path. This section's migration is complete: the old string-taking `Set` is deleted, and all in-repo call sites pass a method expression. +The typed Reminder method handle is implemented in the current code. The +public naming migration to `Reminder` and `NewReminder` remains part of the +0.1.0 API work. The method name is read from the expression once. The table, +poller, and restart recovery use the method-name string. diff --git a/design/transport.md b/design/transport.md index d76dc6d..1f951b8 100644 --- a/design/transport.md +++ b/design/transport.md @@ -1,6 +1,6 @@ # Transport -Moves bytes between nodes. **This layer does not understand message semantics** — it does not know what an Identity, a method, or an entity is. +Moves bytes between nodes. **This layer does not understand message semantics** — it does not know what a GrainId, a method, or a Grain is. ## No gRPC @@ -58,7 +58,7 @@ No mutex protects the pending table. This is not cleanliness for its own sake: b `Send` hands the request together with a reply channel to the owner, then selects on the reply and `ctx.Done()`. -**Each server-side handler runs in its own goroutine; it must not run in the owner.** The owner only registers and hands over; running a handler to completion in the owner would block every other request on this connection behind it — the entire reason correlation ids exist is so requests do not wait on each other. The layer above can least afford this: `gor` packs calls of many entities into one connection, and entity calls are serial anyway, so one busy entity would stall every other entity from the same node. +**Each server-side handler runs in its own goroutine; it must not run in the owner.** The owner only registers and hands over; running a handler to completion in the owner would block every other request on this connection behind it — the entire reason correlation ids exist is so requests do not wait on each other. The layer above can least afford this: `gor` packs calls of many Grains into one connection, and Grain calls are serial anyway, so one busy Grain would stall every other Grain from the same node. Handlers also write responses back to the owner through a channel; only the owner ever lays out frames. @@ -92,7 +92,7 @@ Deciding whether a node is really gone is the membership table's job ([cluster.m ## Encoding is not this layer's business -The transport moves opaque bytes. Encoding happens in the `gor` layer with `encoding/json` — the same story as entity state persistence; no second serialization story is introduced. +The transport moves opaque bytes. Encoding happens in the `gor` layer with `encoding/json` — the same story as Grain state persistence; no second serialization story is introduced. It was chosen not because it is fast but because it is already in the project and humans can read it directly in production. [architecture.md](architecture.md) explains why no custom format, and what that gives up. diff --git a/docs/README.md b/docs/README.md index b708de7..ca24808 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,7 +2,10 @@ This layer describes what `gor` must satisfy: user needs, the API surface, the mental model, the boundaries of responsibility. -Written in product and domain language, for users, assuming the reader does not read the source. Implementation terms (goroutine scheduling, code generation mechanics, storage table structure) belong to [`design/`](../design/README.md). +Written in plain English and in product and domain language, for users who do not read the source. The writing rules are in [writing-style.md](writing-style.md). Implementation terms (goroutine scheduling, code generation mechanics, storage table structure) belong to [`design/`](../design/README.md). + +The root [CONTEXT.md](../CONTEXT.md) is the source for the core product +language. Use its terms before adding a new synonym. ## Annotation conventions @@ -12,8 +15,11 @@ When a document and the implementation diverge significantly, the document gets ## Documents -- [vision.md](vision.md) — positioning, three principles, non-goals, relationship to adjacent approaches. +- [writing-style.md](writing-style.md) — the English and ASD-STE100 writing rule for repository documentation. +- [vision.md](vision.md) — product direction, core promises, and boundaries. - [programming-model.md](programming-model.md) — the programming model and API shape. - [errors.md](errors.md) — call errors, stable error codes, cancellation, and the cross-node boundary. -- [example.md](example.md) — the device-shadow example puts entities, state, scheduled tasks, and cross-entity calls on one runnable usage path. +- [example.md](example.md) — the device-shadow example puts Grains, State, + Reminders, and cross-Grain Calls on one runnable usage path. - [compatibility.md](compatibility.md) — v0 and v1 compatibility promises to users, upgrade boundaries, and known limits. +- [release-0.1.0.md](release-0.1.0.md) — the first announced release contract, supported capabilities, non-goals, and acceptance standard. diff --git a/docs/compatibility.md b/docs/compatibility.md index 7d6a0da..2673b29 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -23,7 +23,7 @@ Within one v0 minor version, patch versions do not actively break documented usa When moving to the next v0 minor version, gor reserves the right to change: -- how the runtime is wired up and entities are called. +- how the Runtime is wired up and Grains are called. - the automatically generated call artifacts. - options, error details, and resource usage. - routing, fault handling, and recovery behavior in a cluster. @@ -37,9 +37,10 @@ Production should pin to a specific v0 minor version. Moving to the next minor v v0's usable scope is single-process. For the published single-process capabilities, users can rely on these basic semantics: -- calls on the same identity execute in order. +- Calls for the same GrainId execute in order. - state confirmed successfully survives a process restart. -- documented scheduled delivery, overload, and failure outcomes behave as described. +- documented Reminder delivery, overload, and failure outcomes behave as + described. These are product promises, not promises about implementation shape, throughput numbers, or exact execution instants. @@ -75,4 +76,4 @@ If this v1 promise must ever be broken, gor will release a new major version wit ## Gap -The readiness work for an announced release is complete (every ROADMAP "required" item is done); staying in 0.0.x is a choice, not a missing requirement. The broader v0 discipline and assembled release notes start at 0.1.0. The README presents single-process as the usable scope and describes multi-node failure detection as direct probing with death voting, while the reliability limits are stated in this document's "What can already be relied on" section. +The pre-announcement checklist is complete, but the first announced release also requires the 0.1.0 product contract's composition and failure-evidence gates. The broader v0 discipline and assembled release notes start at 0.1.0. The README presents single-process as the usable scope and describes multi-node failure detection as direct probing with death voting, while the reliability limits are stated in this document's "What can already be relied on" section. diff --git a/docs/errors.md b/docs/errors.md index d3b3026..a79e5b8 100644 --- a/docs/errors.md +++ b/docs/errors.md @@ -1,6 +1,7 @@ # Errors and cancellation -An entity call may complete on this node or on another. Callers can rely on the same set of outcome rules. Location transparency guarantees exactly that — nothing more. +A Grain Call may complete on this Silo or on another Silo. Callers can rely +on the same outcome rules. Location transparency guarantees exactly that. Location transparency does not preserve in-process error objects. An error's concrete type, fields, wrapping, and implementation details do not become a contract because the call crosses nodes. Calls do not promise exactly-once execution either. diff --git a/docs/example.md b/docs/example.md index a1977b9..3e2fa97 100644 --- a/docs/example.md +++ b/docs/example.md @@ -14,21 +14,29 @@ The example app answers a different question: when would you be glad you used th Chosen because it stresses four things at once — exactly the four reasons `gor` exists: -**Large count, mostly idle.** A hundred thousand devices, but only a few hundred speaking at any moment. The traditional approach keeps all hundred thousand objects in memory, or reads the database on every report. `gor`'s answer: the device itself is an entity — in memory while speaking, gone when silent, with state left in the store. +**Large count, mostly idle.** A hundred thousand devices, but only a few +hundred speaking at any moment. `gor` keeps each Device Grain active while it +speaks and keeps its State in the store when it is idle. **Concurrent writes to one device must be serialized.** The device reports state while operations pushes configuration. When the two collide, without serialization you write a pile of optimistic-lock retries. In `gor` this is the default; users do nothing. -**Offline detection is naturally a scheduled task.** "No report for thirty seconds means offline" — this needs an alarm that follows the device and survives process restarts. That is exactly what the scheduled-task step provides. Polling the whole table for the same job gets linearly more expensive with device count. +**Offline detection is naturally a Reminder.** "No report for thirty seconds +means offline" needs an alarm that follows the Device Grain and survives a +process restart. Polling the whole table gets more expensive as the device +count grows. -**Aggregation needs cross-entity calls.** "How many devices in this workshop are online" requires devices to reach the workshop. This demonstrates entity-to-entity calls, and why the reverse cannot work — the workshop cannot hold references to a hundred thousand devices and ask them one by one. +**Aggregation needs cross-Grain Calls.** Devices notify the Workshop Grain +when their state changes. The Workshop Grain keeps the online GrainIds. It +does not call every Device Grain one by one. ## What the reader should learn In this order, someone new to `gor` should be able to: -1. Recognize what should be an entity — what an identity is, where the boundaries go. +1. Recognize what should be a Grain — what its GrainId is and where its + boundaries go. 2. Know how state is stored, when it is persisted, and what a write conflict does. -3. Attach a scheduled task and know it survives a process restart. +3. Attach a Reminder and know it survives a process restart. 4. Know what state the world is in after a call fails — the point examples most easily gloss over. Point 4 must be written seriously. **No ignored errors in the example.** Every failure is either handled, or a comment explains why it can be ignored here. Examples are copied; copying a `_ = err` copies a bug. diff --git a/docs/programming-model.md b/docs/programming-model.md index 49d3b9b..f956a23 100644 --- a/docs/programming-model.md +++ b/docs/programming-model.md @@ -6,18 +6,23 @@ Only three. -**Entity** — an object with an identity and state. You write a Go struct plus a set of methods. The runtime guarantees calls on the same identity execute serially. +**Grain** — a stateful object with a GrainId. You write a Go struct plus a +set of methods. The Runtime guarantees Calls for the same Grain run +serially. -**Identity** — type + key. `Account("alice")` and `Account("bob")` are two different entities; `Account("alice")` always refers to the same one. No creation, no destruction: it exists from the first call, disappears from memory after enough idleness, state stays in the store, and the next call brings it back. +**GrainId** — a GrainType plus a GrainKey. `Account("alice")` and +`Account("bob")` are two different Grains. `Account("alice")` always names +the same Grain. No create or delete call is needed. The Grain starts at its +first Call, may leave memory after idle time, and keeps State in the store. **Call** — calling a method through an interface. The caller does not know and does not care whether the target is in this process or on another node. -## Declaring an entity +## Declaring a Grain Write the interface first: ```go -//gor:entity +//gor:grain type Account interface { Deposit(ctx context.Context, amount int64) (int64, error) Balance(ctx context.Context) (int64, error) @@ -28,9 +33,12 @@ The first parameter of an interface method must be `context.Context`; the last r ### Generation prerequisite -The `//gor:entity` marker says this interface gets typed calls generated for it. You add the generator to your module once, then run it whenever a marked interface is created or changed, before building. The exact commands and where the generated files land: [../design/codegen.md](../design/codegen.md). +The `//gor:grain` marker says this interface gets typed Calls generated for +it. Add the generator to your module once. Run it when a marked interface +changes, before you build. See [../design/codegen.md](../design/codegen.md). -Every runtime must install the generated output at startup before entities can be registered or references obtained. The startup example below shows where installation happens. +Every Runtime must install the generated output at startup before Grains can +be registered or Grain References can be obtained. Then write the implementation: @@ -55,7 +63,9 @@ func (a *account) Balance(ctx context.Context) (int64, error) { } ``` -The interface, the implementation, and the registration below live together in the entity's package — the factory refers to the unexported `account` type, so the registration cannot be written anywhere else. The startup code calls the registration with the runtime it built: +The interface, implementation, and registration live in the Grain package. +The factory uses the unexported `account` type, so the registration stays in +that package. ```go func Register(rt *gor.Runtime) error { @@ -69,13 +79,13 @@ func Register(rt *gor.Runtime) error { No locks in method bodies, because none are needed — a second call on the same key is never running at the same time. -## The entity knows who it is +## The Grain knows its GrainId -Extending the registration from the previous section — the struct also keeps its identity: +The struct can also keep its GrainId: ```go type account struct { - id gor.Identity + id gor.GrainId balance gor.State[int64] } @@ -89,11 +99,14 @@ func Register(rt *gor.Runtime) error { } ``` -This is needed for logging, for using the key as business data (the `alice` in `Account("alice")` is a username), and for calling another entity and telling it who you are. +The GrainId is useful for logs, business data, and Calls to another Grain. +The `alice` key can be a user name. -An identity is not state. It never enters the store, does not change when the entity is evicted and reactivated, and does not roll back on write conflicts. When the same identity is active on two nodes at once, both activations' `id` is the same value. +The GrainId is not State. It is not stored as Grain State. It does not change +when the Grain leaves memory and starts again. Two Activations for one +GrainId have the same GrainId. -## The entity reads time +## The Grain reads time The `Binder` is given to the factory once, at activation. If method bodies need it, keep it in the factory — registration shaped as in the previous sections: @@ -116,9 +129,10 @@ func (d *device) Report(ctx context.Context, value float64) error { } ``` -Do not use `time.Now()`. Time read by an entity must come from the runtime — tests need to control it, and in simulation each node's clock can carry a different offset. This is the same rule the library itself follows. +Do not use `time.Now()`. Time read by a Grain must come from the Runtime. +Tests must control time, and a future Silo may have a different clock. -## One entity calls another +## One Grain calls another The same function as calling from outside, with a different first argument: @@ -126,18 +140,22 @@ The same function as calling from outside, with a different first argument: gor.Ref[Workshop](d.b, workshopID).DeviceOnline(ctx, deviceID) ``` -Outside, you hold the runtime; inside, the `Binder`. An entity does not capture a runtime object to call others — the factory signature is `func(b *gor.Binder) T`, and that one parameter is enough. +Outside, the caller holds the Runtime. Inside, the Grain holds the `Binder`. +The factory needs only `func(b *gor.Binder) T`. -Cross-entity calls are the most common thing virtual entities do. They must be as easy as local method calls, or users will pile logic into one giant entity to avoid them. +Cross-Grain Calls are part of the virtual Grain model. They use the same +typed reference as a local Call. -## Calling an entity +## Calling a Grain ```go acct := gor.Ref[Account](rt, "alice") balance, err := acct.Deposit(ctx, 100) ``` -`acct` has type `Account`. A wrong argument type or a nonexistent method is a compile error. This is the key difference from `any`-based APIs; the price is running code generation once, see [../design/codegen.md](../design/codegen.md). +`acct` has type `Account`. A wrong argument type or a missing method is a +compile error. This is the key difference from `any`-based APIs. See +[../design/codegen.md](../design/codegen.md). ### Cluster calls and deployment limits @@ -153,33 +171,59 @@ Arguments and return values go through JSON across nodes, so they must be JSON-e ## Call outcomes and ordering -One entity processes calls in a queue. When the queue is full, new calls are rejected for overload outright: the method never starts, and state does not change. +One Grain processes Calls in a queue. When the queue is full, new Calls are +rejected for overload. The method does not start, and State does not change. -Timeout or cancellation only means the caller stopped waiting. The method may have started, may even have changed state; a cross-node call hitting a post-send network error is the same. Callers cannot tell from this error whether the method ran. Do not retry these two outcomes as if they were overload rejections. +Timeout or cancellation means that the caller stopped waiting. The method +may have started and may have changed State. A delivery error after a Call +was sent has the same unknown result. The caller cannot know if the Business +Action ran. -A method panic makes the call return an error and discards the current instance. Calls already queued but not started also end in error; they are not rerun on a fresh instance. The next call rebuilds the instance from persistent state. +A method panic returns an error and discards the current Activation. Queued +Calls that did not start also return errors. The Runtime does not replay +them. The next Call builds a new Activation from confirmed State. -While an entity handles one call, it does not start a second. A call that would close a cycle — A calling B and B calling A back — is detected along the call chain and fails with an error that names the entities in the cycle, instead of hanging until the caller gives up; its stable code tells it apart from an ordinary timeout. The runtime does not retry automatically: whether retrying is safe and how to avoid duplicate business actions is the caller's judgment. +While a Grain handles one Call, it does not start a second Call. A Call cycle +is detected and fails instead of waiting forever. The Runtime does not retry +the Call. The Application decides whether a Safe Repeat is valid. -Calls from one caller to one entity, sent locally in sequence, execute in issue order. Cross-node, that order is not guaranteed; operations with ordering dependencies must express the dependency in business data, not rely on network arrival order. +Calls from one caller to one Grain, sent locally in sequence, execute in issue +order. A future cluster does not promise network arrival order. ## State -`gor.State[T]` carries state. `Get()` reads the current in-memory value; `Set()` writes and persists it. +`gor.State[T]` carries State. `Get()` reads the current value. `Set()` writes +and persists it. -An entity can have several cells, distinguished by name. They are stored as one record, so any cell write updates the whole entity's version. +`Exists()` tells whether confirmed State is present. It is different from +reading a present value that contains the type's zero value. `Clear()` +removes confirmed State. After `Clear()` succeeds, the next Activation sees +the State as absent. -**When a cell holds a map or slice, `Get()` returns that very instance, not a copy.** Mutating it only counts after `Set()` — mutate without writing, and the value changes in memory but not in the store; after eviction and return, the entity reverts to the old value. Copy-before-mutate is a style choice; persistence depends only on `Set()`. +A Grain can have several named State values. They are stored as one Grain +record, so any State write updates the Grain version. -Every `Set()` tries to persist immediately. Only success makes the value the current persisted value; on failure the last confirmed value is kept and the current instance is discarded. After an error, do not assume the instance is still usable; the next call reads state back. +**When State holds a map or slice, `Get()` returns that instance, not a copy.** +The change is persisted only after `Set()`. After the Grain leaves memory, an +unsaved change is lost. -Multiple `Set()` calls in one method are not a transaction. An earlier write may have succeeded while a later one fails; when the business result must be atomic, the business must organize the related data into one state update. +Every `Set()` tries to persist immediately. Only success confirms the new +value. On failure, the Runtime keeps the last confirmed value and discards +the current Activation. The next Call reads State again. -State must be JSON-encodable. The runtime does not carry applications through state-structure evolution; field additions, removals, or format changes are the application's job — read old formats, write new ones. +Multiple `Set()` calls in one method are separate State writes. An earlier +write may succeed before a later write fails. Keep one business change in one +State update when that result is required. -Concurrency semantics, stated plainly: in cluster mode, the runtime does not guarantee that only one `Account("alice")` runs in the whole world at any moment. A double-activation window opens whenever the cluster's membership is changing — nodes joining, leaving, failing, or being partitioned — and closes once every node's view of the membership agrees. While it is open, two nodes may each have the same entity active and both accept a write to it; `Set()` carries an optimistic-concurrency check, so the write that lands second fails instead of silently overwriting the first. That failure is returned to the caller, who must retry; the runtime does not retry it. A call that always succeeds on a single node can therefore return an error on a cluster during this window — not because the work was wrong, but because a second activation raced it. +State must be JSON-encodable. The Application owns State format changes. -This is not implementation laziness — Orleans' default directory has the same semantics, and its official docs say so (see [../research/orleans-internals.md](../research/orleans-internals.md) (in Chinese)). In single-node mode this window does not exist, so this failure does not occur there. +In a future cluster, the Runtime may have two Activations for one Grain while +ownership changes. Both may accept a Call. The State version check rejects +the old write instead of silently replacing newer State. The caller receives +a conflict and decides whether to retry. + +This behavior follows the Orleans model. A single Silo has no ownership +change, so this cluster conflict does not occur there. ## How durable a state write is @@ -192,76 +236,104 @@ Two levels: The trade is throughput. Forcing every write to disk costs time; most services can tolerate losing the most recent changes after a hard crash, and Relaxed lets those services change state faster. -Relaxed touches state and nothing else. Scheduled tasks still fire at most once after a crash; if you run more than one node, the bookkeeping the nodes use to agree on who owns what is unaffected. +Relaxed touches Grain State and nothing else. Reminders still fire at most +once after a crash. Future cluster ownership data is unaffected. If you do not choose, you get Full. The mechanism behind the trade and its exact limits are in the [persistence design](../design/persistence.md). -## Scheduled wake-up +## Reminder -State connects to the store via `gor.State[T]`; scheduled tasks take a cell from `b` the same way: +State connects to the store via `gor.State[T]`; a Reminder uses the Binder in +the same way: ```go type account struct { balance gor.State[int64] - schedule gor.Schedule[Account] + reminder gor.Reminder[Account] } func (a *account) Open(ctx context.Context) error { - return a.schedule.Set(ctx, "monthly-interest", gor.Every(30*24*time.Hour), gor.Handle(Account.ApplyInterest)) + return a.reminder.Set(ctx, "monthly-interest", gor.Every(30*24*time.Hour), gor.Handle(Account.ApplyInterest)) } -func (a *account) ApplyInterest(ctx context.Context) error { ... } +func (a *account) ApplyInterest(ctx context.Context, tick gor.TickStatus) error { ... } ``` -Scheduled tasks are persistent: after a process crash, a task that has come due still fires. If the object is not in memory when the task comes due, it is woken up. +A Reminder is persistent. After a process crash, a due Reminder can still +run. If the Grain is not in memory, the Runtime starts its Activation. -The schedule is typed to the entity's interface, and the wake-up method is named by a method expression — a typo or a rename is a compile error, not a failure hours later at delivery. What comes due is still a method name, not a function value — after a crash nobody can restore a closure; only the name can be stored. The invoked method takes only `ctx` and returns only `error`. +The Reminder is typed to the Grain interface. The Reminder method uses a +method expression, so a typo or rename is a compile error. The Runtime stores +the method name, not a function value. The method takes `ctx` and +`gor.TickStatus`, and returns `error`. -It is not `time.AfterFunc`: do not expect millisecond precision, and do not expect missed firings during downtime to be made up (it fires once on return, then moves on). +It is not `time.AfterFunc`. It does not promise millisecond precision. It +does not replay every tick missed during downtime. -One object has at most one task per name; setting the same name again reschedules it. +One Grain has at most one Reminder with a given name. Setting the same name +again changes that Reminder. -Tasks can be one-shot or periodic; after cancellation they are not kept. A one-shot task is delivered at most once when due, then disappears. +A Reminder can be one-shot or periodic. Cancellation removes it. A one-shot +Reminder is delivered at most once when due. -Scheduled wake-up promises at-most-once delivery, not exactly-once method execution. The system confirms that the due time was claimed, then delivers the method; a crash between the two can miss this firing. Failed methods are not retried automatically either; the error still goes to the error sink below. +A Reminder promises at-most-once delivery. It does not promise exactly-once +method execution. The Runtime claims the due time before delivery. A crash +between these actions can miss the Call. A failed method is not retried; its +error goes to the background error sink. -A state change and setting, rescheduling, or canceling a scheduled task are not one atomic business operation. Either side can succeed alone; business semantics that need both must handle this window in the application. +A State change and a Reminder change are separate Runtime actions. The +Application must handle a partial result when both actions are needed. -## How an entity starts and leaves +## How a Grain starts and leaves -An entity can initialize after it starts serving; when initialization fails, that call fails and the next call rebuilds the entity. +A Grain can initialize when its Activation starts. If initialization fails, +that Call fails and the next Call builds a new Activation. -An entity can do final teardown before leaving. It learns whether this leave is due to idleness, the current node no longer owning it, a graceful stop, or the instance no longer being trusted. The application can then tell apart "reclaim local resources", "hand back node ownership", "teardown before process exit", and "handle as a fault". +A Grain can run a deactivation hook before it leaves. The hook receives the +reason: idle, ownership lost, normal shutdown, or an untrusted Activation. -Teardown cannot prevent the entity from leaving. A graceful stop waits for teardown that has already started to return; teardown should finish promptly. The hook gets a fresh work context with no deadline that is never canceled. Under an abrupt stop or when the node is declared dead, teardown that has not started does not run; teardown that has started is not force-aborted. +The hook cannot prevent deactivation. A graceful stop waits for a hook that +has started. An abrupt stop does not start new hooks. A hook that has started +is not force-aborted. ## Failures nobody is waiting for -Two application actions can fail with no caller waiting for the result: a claimed scheduled delivery fails, or teardown before the entity leaves fails. The runtime can be configured with a background error sink. +Two Application actions can fail with no caller waiting: a claimed Reminder +Call can fail, or a deactivation hook can fail. The Runtime can send both to +a background error sink. -Each event gives the entity identity, the original error, and a clear source. A scheduled delivery gives the delivered action's name; a teardown failure gives the reason the entity left. Sources are not application-conventioned text, so an action name that happens to equal the teardown name cannot be confused with it. +Each event gives the GrainId, the original error, and a source. A Reminder +event gives the method name. A deactivation event gives the leave reason. Errors still follow the [Errors and cancellation](errors.md) section. Across nodes, only declared stable codes are usable for business branching; error text is for display and logging. -The sink does not retry, back off, or alert for the application. Scheduled delivery is at-most-once by design; an application that retries must design idempotency and state itself. Poller scan and claim failures are not reported here either. +The sink does not retry, back off, or alert. Reminder delivery is at-most-once +by design. The Application owns any Safe Repeat behavior. -When migrating an existing application, change the handler that used to receive identity, action name, and error to receive an event, then read the action name or the leave reason from the source. Stop guessing the source from the action name. +The handler must read the Reminder method or deactivation reason from the +event source. It must not infer the source from a method name. ### Gap -The background error sink is implemented: each event gives the entity, the original error, and a closed set of sources; scheduled delivery carries the delivered action's name, teardown failure carries the entity's leave reason, sources branch by type instead of text comparison, and the source set cannot grow outside the runtime. Deactivation reasons are implemented: when an entity leaves, it receives one of four reasons — idle, current node lost ownership, graceful stop, or instance untrusted; the reason is fixed when the leave begins and later events never rewrite it; the work context given at leave has no deadline and is never canceled. A graceful stop waits for teardown that has started; abrupt stops and declared-dead nodes skip teardown that has not started and do not wait for teardown that has. Poller scan and claim failures are not reported from this sink; the delivery canceled mid-shutdown is not reported either. Everything else in this section is implemented. +The background error sink and deactivation reasons are implemented. The +remaining release work is listed in [../ROADMAP.md](../ROADMAP.md). ## Runtime observability -The runtime hands the application two kinds of facts. First, a snapshot of this node's current activations: which entities are serving, and how many queued, not-yet-started calls each has. It observes only this node; it does not aggregate for the cluster. +The Runtime provides two kinds of facts. First, it provides a snapshot of +this Silo's active Activations and their queued Calls. It does not aggregate +data for a future cluster. -Second, an event per completed call. The event gives the result the caller saw, the duration, and the target's type and method. When the caller cancels, the event records the cancellation as the result; even if the method later runs to completion, there is no second event. A cross-node call is recorded once, at the initiating node; the receiving node does not record it again. +Second, it provides one event for each completed Call. The event gives the +caller result, duration, GrainType, and method. A canceled Call has one +canceled result even if the method later completes. -Completion-event callbacks run synchronously with the caller. A callback must not block or do I/O — the delay would land on the caller's own result. The runtime does no aggregation, export, or alerting of monitoring data; applications wire it into existing systems. +Completion callbacks run with the caller. A callback must not block or do +I/O. The Runtime does not aggregate, export, or alert these events. ## Runtime startup -Single node, state in a local file: +Single Silo, State in a local file: ```go if err := os.MkdirAll("data", 0o755); err != nil { return err } @@ -276,7 +348,8 @@ defer rt.Close() if err := gorgen.Install(rt); err != nil { return err } ``` -`Install` hands the generated proxies and dispatch functions to the runtime. Without this line, `Register` and `Ref` fail at startup — not at the first call. +`Install` hands generated proxies and dispatch functions to the Runtime. +Without this line, Grain registration and Grain References fail at startup. A cluster must explicitly hand the runtime the state store, the shared membership table, this node's address, this startup's generation, and the transport: @@ -298,11 +371,12 @@ if err != nil { defer rt.Close() ``` -All nodes share `memberStore`; `generation` must be a fresh value on every rejoin at the same address. `Runtime.Close` closes the configured transport. The difference between single-node and cluster is configuration, not business code. +All nodes share `memberStore`; `generation` must be a fresh value on every rejoin at the same address. `Runtime.Close` closes the configured transport. The difference between a single Silo and a future cluster is configuration, not business code. ## The runtime can stop itself -In a cluster, a node can be declared dead by others. After that it serves no entity — serving with an identity the whole world believes dead only writes data nobody will ever see. +In a future cluster, a Silo can be declared dead by other Silos. After that it +must serve no Grain. So the runtime provides a signal: @@ -310,15 +384,22 @@ So the runtime provides a signal: <-rt.Done() // closed, or declared dead ``` -When it closes, the runtime also stops admitting new entity calls. Calls issued after that — from this process or another node — get a reliably identifiable stop error. Graceful stops and abrupt stops use the same stop error. When the cluster declares the node dead, the error says the node stopped serving. Codes and how to check them: [errors.md](errors.md). +When it closes, the Runtime stops admitting new Grain Calls. Calls issued +after that get a stable stop error. Codes and checks are in [errors.md](errors.md). -The stop signal does not rewrite results admitted earlier. A graceful stop lets started methods finish, rejects queued calls that have not started, and waits for methods and deactivations to end. An abrupt stop and a death declaration cancel started methods and reject the queue, but cannot force-abort user code that ignores cancellation. An admitted call may still finish after the signal closes; a call issued after the signal closes cannot succeed. +The stop signal does not rewrite results for Calls already admitted. A +graceful stop lets started methods finish and rejects queued Calls. An abrupt +stop cancels started methods but cannot force-abort user code that ignores +cancellation. Your process should exit, or build a new runtime and rejoin. Ignoring the signal does not silently break anything, but the service should not keep advertising itself as available. ### Gap -This section's admission boundary is implemented: the stop transition is the linearization point of admission; after it, calls from this process or another node get the corresponding stable stop error (`gor.runtime_closed` or `gor.node_dead`). "The stop signal does not rewrite previously admitted results" holds only for graceful stops; `Kill` and death declarations cancel admitted calls, and their results become cancellation errors. +The Runtime admission boundary is implemented. Calls after stop receive the +stable stop error. Calls admitted before stop keep the result defined by the +stop mode. The code surface still needs the public Grain terminology and the +State and Reminder operations described in this target API. ## Mental model comparison @@ -326,9 +407,10 @@ If you have used other systems: | Concept | Orleans | Temporal | Restate | gor | |---|---|---|---|---| -| Stateful object with an identity | Grain | — | Virtual Object | Entity | -| Identity | GrainId | WorkflowId | Object Key | Identity | +| Stateful object with a GrainId | Grain | — | Virtual Object | Grain | +| GrainId | GrainId | WorkflowId | Object Key | GrainId | | Persistent state | `[PersistentState]` | Workflow variables | built-in K/V | `State[T]` | -| Scheduled wake-up | Reminder | Timer | — | Schedule | +| Scheduled action | Reminder | Timer | — | Reminder | -The table only builds intuition. Semantics are not fully equivalent; notably, Temporal workflows have a deterministic-replay constraint that `gor` does not — `gor` recovers from persisted state, not by replaying an event log. +The table only builds intuition. Semantics are not fully equivalent. The +Orleans Grain model is the reference for `gor`. diff --git a/docs/release-0.1.0.md b/docs/release-0.1.0.md new file mode 100644 index 0000000..7fe1732 --- /dev/null +++ b/docs/release-0.1.0.md @@ -0,0 +1,154 @@ +# 0.1.0 Product Contract + +This document states what gor 0.1.0 promises. It is not a status report. +[ROADMAP.md](../ROADMAP.md) states what is done. The [design +document](../design/release-0.1.0.md) states how the release is delivered. + +## Release promise + +gor 0.1.0 is a reliable single-Silo Grain Runtime for Go Applications. It +starts Grains when Calls need them, serializes Calls for each Grain, and +keeps confirmed State in local storage. + +The release targets one process and one local store. It does not require a +network or a cluster. + +The release is ready only when users can understand the rules, test failure +cases, and use the public API without private runtime code. + +## Supported capabilities + +### Grains and Grain References + +A Grain is identified by a GrainType and a GrainKey. Together they form a +GrainId. + +A Grain Reference names a Grain without starting it. The first Call can +start its Activation. An idle Grain can leave memory. A later Call can start +it again and load its State. + +Calls for one Grain run one at a time. Calls for different Grains may run at +the same time. The Runtime defines results for overload, timeout, cancel, +start failure, method failure, panic, and shutdown. + +### Calls + +The public API uses typed Go interfaces. A caller uses a typed method call. +It does not send an untyped message. + +A Grain may call another Grain through a Grain Reference. Local and future +remote Calls use the same model. + +The Runtime does not re-enter a Grain during a Call. Reentrant and +interleaved Calls are outside 0.1.0. A call cycle fails instead of waiting +forever. + +The Runtime does not retry a Business Action after an unknown result. The +Application must decide whether a Safe Repeat is valid. + +### Request Context + +A Call may carry Request Context, such as a trace ID. The called Grain can +read it during the Call. The Grain Runtime does not save it. + +### Persistent State + +A Grain can own named State values. A successful State write becomes the +confirmed value for the Grain. + +State provides these user-visible operations: + +- Read the current value. +- Write a new value. +- Check whether a value exists. +- Clear the value. + +An absent value is different from a value that contains the type's zero +value. Clear removes the confirmed value. A later Activation observes that +the value is absent. + +A version check stops an old Activation from replacing newer State. The +Runtime reports a conflict. + +State has two durability levels: + +- **Full**: confirmed writes are on disk before the Call returns. +- **Relaxed**: a hard machine failure may lose recent confirmed writes. + A normal process restart keeps confirmed writes. + +State is Application data. The Application owns its meaning and its format +changes. + +### Reminders + +A Grain can set a named Reminder. A Reminder can run once or repeat on a +period. The setting, a new setting, and cancellation survive a normal +process restart. + +A due Reminder can start a Grain that is not in memory. The Runtime claims a +Reminder before it delivers the Call. A failure after the claim can miss +that delivery. The Runtime does not retry the Call automatically. + +A periodic Reminder reports its first tick time, period, and current tick +time to the Grain. A late process does not receive every missed tick. + +An Application that needs recovery must save a pending Business Action and +use a Safe Repeat handler. + +### Lifecycle and background errors + +A Grain can run code when its Activation starts and when it leaves memory. +The leave reason tells the Application why the Activation ended. + +Failures with no waiting caller go to the configured background error sink. +Examples include a failed Reminder Call and a failed deactivation hook. The +sink reports the original error and its source. It does not add hidden +retries. + +### Observability + +The Runtime provides a snapshot of this Silo's active Activations and a +completion event for each Call. The Application chooses its own metrics, +traces, storage, and alerts. + +## Single-Silo boundary + +The 0.1.0 release is a single-Silo product. The single-Silo path must not +need a network, a remote service, or cluster membership. + +GrainId, Grain Reference, Call, State, Reminder, and encoding boundaries +must leave room for future cluster work. This is a design rule. It is not a +promise of reliable cluster operation in 0.1.0. + +Multi-Silo operation, ownership changes, network failure handling, rolling +upgrades, and cluster administration are outside the 0.1.0 promise. + +## Non-goals + +0.1.0 does not provide Call Filters, reentrant or interleaved Grain Calls, +cluster operation tools, incompatible rolling upgrades, cloud storage, or +unlimited scale. + +These limits keep the single-Silo product small and reliable. + +## Acceptance standard + +The release is ready only when all items below are true: + +1. A small example can define a Grain, get a typed Grain Reference, write + and clear State, and set a Reminder through the public API. +2. The example can stop and start the process without private runtime calls + or manual database repair. +3. Deterministic tests cover activation, serialized Calls, State conflicts, + State clearing, process failure, Reminder claims, Safe Repeats, cancel, + shutdown, and background errors. +4. The docs state the result for timeout, cancel, delivery failure, and a + claimed Reminder whose Call did not run. +5. The full repository test gate passes. It includes unit tests, simulation, + generated-code tests, network tests, lint, and race tests. + +## Gap + +The single-Silo Runtime, State, Reminders, typed Calls, lifecycle, and +observations already exist in parts. The 0.1.0 work is not complete until +these parts use the public Grain model together under restart and failure. diff --git a/docs/vision.md b/docs/vision.md index 83f03c1..8bd5a69 100644 --- a/docs/vision.md +++ b/docs/vision.md @@ -2,71 +2,85 @@ ## In one sentence -Objects in Go programs that have an identity, hold state, and keep running after a crash should be as easy to write as ordinary structs. +`gor` is a Go port of the Orleans runtime model for programs that need +stateful Grains on one machine. -## The problem +## The product direction -People writing stateful services solve the same batch of problems over and over: who owns this user's data right now? Two requests change it at once — what happens? The process dies — what about everything in memory? A scheduled task — will it still fire after a restart? +`gor` gives a Go program a Grain Runtime. A Grain has a GrainType, a +GrainKey, State, and behavior. The runtime starts a Grain when a Call needs +it. It keeps Calls for one Grain in order and keeps confirmed State in local +storage. -The usual answer: push state into a database, reread it on every request, add locks or optimistic concurrency, and run scheduled tasks as a separate system. The code is full of "read, check, write, handle the conflict". +The first product is one Silo on one machine. It must be useful without a +network, a sidecar, or a remote service. A later cluster extension may move +Grain ownership between Silos. It must not change the Grain model. -`gor`'s answer: seal all of that inside the runtime. You declare an object type and give it a key; the runtime guarantees calls on the same key execute one after another, state survives between calls, and state survives process restarts. You write method bodies, not locks. +## Core promises -## The direction +### Orleans model -The direction is narrow and absolute: **a program that needs stateful objects should get them as easily on one machine as it gets a map — and clustering, if it ever matters, is something added later, never the price of getting started.** +`gor` uses the Orleans model and terms. A Grain Reference names a Grain +without starting its Activation. The first Call can start the Activation. +The Grain may leave memory later. Its GrainId and confirmed State remain. -Most systems that need stateful objects never grow to dozens of machines. But the existing options all make you pay the distributed price first: install a server, configure a database, run a sidecar. `gor` is the reverse: `import` it and it works, state lands in an embedded store, and one binary is a complete system. Clustering is an optional extension, not the price of admission. +The Go API may use Go forms where the languages differ. The runtime meaning +must stay aligned with Orleans unless a product spec states a difference. -This is not a "start small, scale later" sales line. It is a claim about what the project is *for*. If a user cannot run `gor` on a single node and trust it, the project has failed, regardless of what the cluster can do. Everything else the project spends effort on exists to make that one claim true and trustworthy. Two commitments carry the weight. +### Reliable single Silo -### Commitment one — types, not `any` +The single-Silo product must make these results dependable: -Comparable Go implementations are typically `Ask(target, message any) (any, error)`. That pushes every error the compiler could catch to runtime: wrong message type, a forgotten case, a renamed field — all of it only shows up when you run. +- Calls for one Grain run one at a time. +- State survives a normal process restart. +- State writes report conflicts instead of silently replacing newer State. +- Reminders survive a normal process restart. +- Lifecycle and background failures are visible to the Application. +- A hard failure does not silently report an unknown result as success. -`gor` requires method signatures to be Go interfaces, so type checking at the call site is exactly like calling a local method. The price is a code generation step, and we accept that price. This commitment exists so the single-node experience is "call a method," not "send a message and pray." +The product must state loss and retry limits in user language. It must not +hide them in implementation details. -### Commitment two — reproducible tests, not hope +### Typed Calls -Distributed system bugs concentrate in timing: this message arrived late, this node died at that instant, these two events happened in the wrong order. Testing with real networks and real time only catches the part that happens to show itself when you are lucky. +Applications define Grain interfaces in Go. Generated code gives callers a +typed Grain Reference. A wrong argument type must fail at compile time. -`gor` has required from day one: all I/O behind interfaces, all time injectable, all components explicit state machines. That way one failure reproduces exactly from one seed. This constraint keeps shaping every design decision — it determines whether this project is worth trusting more than any single feature. It is the reason a single-node user can believe "state survives a crash" without running the crash themselves. +### Reproducible behavior -## Where clustering fits +Tests must control time and I/O through explicit boundaries. Failure tests +must use a seed that can reproduce the same decisions. This rule applies to +the single-Silo product and to the future cluster extension. -Clustering is an optional extension. It exists, it is shipped, and it is not going to be removed. But it is not the main line, and the project does not ask single-node users to pay for it. +## Future cluster boundary -What clustering buys: the same objects, running on more than one machine, with calls routed to whichever node currently owns a key. For a workload that has outgrown one machine, that is the path. +Cluster support is a later extension. It may add several Silos, shared Grain +ownership, routing, and transport. The 0.1.0 product does not promise these +features. -What clustering costs, stated plainly: a failure mode that single-node never has. While the nodes are still settling who owns what — nodes joining, leaving, failing, or split by a network problem — two nodes can each hold the same object active at once. Both accept a write; the one that lands second fails and is returned to the caller as a conflict. On a single node this cannot happen: there is only ever one copy of an object, and calls on it run one at a time. Measured multi-node operation surfaces this: about one in eight test rounds returned at least one write as a conflict that a single node never returns. The caller must retry; the runtime does not. +The public model must leave room for this extension. A Grain Reference must +not depend on a local memory address. State must keep a version that can +reject an old write. These are design constraints for future work, not +cluster promises in 0.1.0. -That is the honest shape of the trade. A single-node user never meets it. A cluster user meets it whenever ownership is changing, and absorbs it. `gor` does not pretend to do better than this; Orleans' default placement has the same shape. +## Product boundaries -## Non-goals +The 0.1.0 product does not provide: -- **No Orleans compatibility layer.** The inspiration is Orleans, but its API shape carries .NET traces (`Task`, `AsyncLocal`, version-tolerant serialization); carrying them into Go is a burden. When concepts do not match, use a different name. -- **No general-purpose actor framework.** Supervision trees, mailbox policies, behavior switching, actor hierarchies — that is Akka's territory. `gor` does exactly one thing: persistent objects that execute serially by key. -- **No workflow DSL.** No orchestration graphs, no saga syntax. Users write business logic in ordinary Go control flow. -- **No cross-entity transactions.** A call that touches two objects and fails halfway fails halfway — no rollback, no outbox. If two pieces of state must change together, make them one object. -- **No unbounded scaling.** The target is one machine, with a small cluster as an optional extension. Beyond that, use Temporal. +- Call Filters. +- Reentrant or interleaved Grain Calls. +- Cluster operation or cluster administration tools. +- Rolling upgrades between incompatible Application versions. +- Unbounded scale. -## Relationship to adjacent approaches +These limits keep the first public release small and reliable. Later work +may add a capability only after its behavior and failure rules are defined. -In the same problem space, `gor` occupies the "library" cell: +## Related systems -- **Temporal** is the most mature product in this space, but it is a system to deploy (server + database + workers). It fits teams where workflows are the business core. -- **Restate / Rivet** are closest in form to the ideal (single binary), but they are Rust servers; the Go side is only an SDK client. -- **Dapr** has virtual actors, but it is a sidecar model — one more deployment unit and one more network hop. -- **goakt** is the closest library in Go, but its API is `any` in, `any` out, and it has no first-class support for persistent state. +`gor` is a library. It runs inside the Application and uses a local store. +It does not require a separate runtime service for the single-Silo product. -Measured data: [../research/landscape.md](../research/landscape.md) (in Chinese). - -## Inspiration and divergence - -Orleans' virtual actor model solves a real problem: no explicit create or destroy — a key reference is enough, and the runtime handles activation. This model deserves to be carried over. - -But two things deserve honest record: - -**Orleans itself is moving elsewhere.** Orleans 10 added journaling and durable jobs; its founder has moved to Temporal and publicly stopped using the word "actor". The market consensus has drifted from "virtual actors" to "durable execution". `gor` follows that consensus — the selling point is "keeps running after a crash", not "actor model". - -**Projects in this spot have died.** Orbit is EA's JVM virtual-actor implementation, inspired by Orleans: 1724 stars, rewritten in Kotlin once, abandoned in 2021. It lost to no one technically; it lost in the ecosystem. Reminder: differentiated positioning matters more than feature completeness. +The project uses Orleans as its model reference. It does not promise source +or binary compatibility with Orleans. It promises the Orleans Grain model in +a Go API. diff --git a/docs/writing-style.md b/docs/writing-style.md new file mode 100644 index 0000000..edd3850 --- /dev/null +++ b/docs/writing-style.md @@ -0,0 +1,85 @@ +# Documentation Language and Style + +This document is the writing rule for gor documentation. + +## Standard + +Use [ASD-STE100 Simplified Technical English, Issue +9](https://www.asd-ste100.org/). It has writing rules and a controlled +dictionary. It selects simple words and gives each word a clear meaning. + +Use [ISO 24495-1:2023 Plain +language](https://www.iso.org/standard/78907.html) as a reader-focused guide. +The reader must be able to find, understand, and use the needed information. + +This project does not claim formal compliance with every ASD-STE100 +dictionary rule. New and changed prose must follow the ASD-STE100 writing +rules. Project terms such as `Grain`, `Activation`, and `CAS` are technical +terms. Define them before use. + +Use American English spelling. + +## Scope + +This rule covers all new and changed prose in: + +- `docs/`; +- `design/`; +- `README` files; +- `ROADMAP.md`; +- public API comments and examples. + +Use English only. Code, names, commands, logs, URLs, and quoted source text +are not prose. Do not add Chinese translations next to the English text. + +Use the terms in the root [CONTEXT.md](../CONTEXT.md). Add a new term only +when the current language cannot express the domain. Add the term to +`CONTEXT.md` when the project accepts it. + +## Writing rules + +- Put the subject and the verb near the start of a sentence. +- Use active voice. Say who does the action. +- Put one topic in each sentence. +- Use a common word when it has the needed meaning. Use `start`, not `begin`, + `commence`, or `initiate`. +- Use one word for one meaning in one document. +- Use a technical noun or verb only when it is needed. Define it before use. +- Avoid long noun groups and abstract nouns when a direct verb works. +- Avoid idioms, jokes, slogans, marketing language, and vague claims. +- Avoid an `-ing` form when it can make the sentence unclear. +- Use `must` for a requirement, `may` for an option, and `must not` for a ban. +- State the user-visible result before the implementation detail. +- Use lists and tables when they make repeated facts easier to scan. +- Keep examples short, correct, and consistent with the public API. + +Target sentence length is 20 words or fewer. Avoid sentences longer than 30 +words unless the sentence is a code rule, a table cell, or a necessary exact +definition. + +## Product and design documents + +`docs/` uses product and domain language. It explains what a user can do and +what result the user can expect. + +`design/` may use technical terms. It must still use plain sentences, and it +must define a term when a new reader may not know it. + +Both layers must state failure behavior. Do not hide a retry rule, an +ordering rule, or a data-loss limit in an example. + +## Review gate + +Before merging a documentation change, check: + +1. Is every prose sentence in English? +2. Can a new reader tell what to do and what will happen? +3. Is each special term needed and defined? +4. Are requirements and options written with `must`, `may`, and `must not`? +5. Are sentences short enough to read without parsing them twice? +6. Do examples and links point to real, current content? + +Use a readability score as a warning, not as proof of clear writing. Aim for +Flesch-Kincaid Grade 9 or lower in `docs/` and Grade 11 or lower in `design/`. +When technical terms raise the score, shorten the nearby sentences and define +the terms. diff --git a/examples/shadow/README.md b/examples/shadow/README.md index f837549..03a26e8 100644 --- a/examples/shadow/README.md +++ b/examples/shadow/README.md @@ -2,21 +2,26 @@ A directly runnable device-shadow service: devices report state, the service keeps the last report; after more than 30 seconds without a new message, the device goes offline and its workshop's online count updates. -## Why these things are entities +## Why these things are Grains -In the example, `Device` and `Workshop` both have stable identities, and each owns a set of state that must be updated serially. +In the example, `Device` and `Workshop` are Grain types. Each Grain has a +stable GrainId and State that must be updated serially. The four points below all correspond to the domain code in `domain/domain.go`. -- **Large device count, mostly idle.** `Device` uses the device id as its identity, with state in `gor.State`; the runtime can evict idle activations from memory, while state stays in the store. -- **Writes to one device must be serialized.** Reports and configuration both change the device's own shadow directly, with no locks; calls on the same identity are queued and executed by `gor`. Readers can see both entry points directly on the `Device` interface. -- **Offline is a one-shot scheduled task that follows the device.** Every report resets the `offline` alarm; when the task fires, it marks the device offline and notifies the workshop. -- **Online count is a cross-entity aggregation.** Devices notify `Workshop` proactively on going online, changing workshops, and going offline; the workshop only keeps the identity set of online devices — it does not hold device references and query them one by one. - -These are not about splitting the code into more types; each identity needs its own activation, state, and serialized calls. - -## What gor does not provide - -Shadow write, offline-alarm reset, and workshop notification are three independent operations. `gor` provides no cross-entity transactions and no compensation or rollback for the user: if the shadow write succeeds and a later alarm or notification fails, the device and workshop can be temporarily inconsistent. The example returns the error to the caller and leaves the window for the business to decide whether it is acceptable; the implementation order is in the `Report` method of `domain/domain.go`. +- **Large device count, mostly idle.** `Device` uses the device key as its + GrainKey. Its State uses `gor.State`. The Runtime can evict idle + Activations while State stays in the store. +- **Writes to one device must be serialized.** Reports and configuration + change the Device Grain without locks. Calls for the same GrainId are + queued and executed by `gor`. +- **Offline is a one-shot Reminder.** Each report resets the `offline` + Reminder. When it runs, it marks the Device Grain offline and notifies the + Workshop Grain. +- **Online count is a cross-Grain aggregation.** Devices notify `Workshop` + when they go online, change workshops, or go offline. The Workshop Grain + keeps GrainIds. It does not call every Device Grain one by one. + +Each GrainId has its own Activation, State, and serialized Calls. ## Running it @@ -26,7 +31,7 @@ Run from the gor repository root: go run ./examples/shadow/cmd/shadow ``` -Registering the shadow entities only needs the runtime: +Registering the shadow Grains only needs the Runtime: ```go if err := shadow.Register(rt); err != nil { @@ -34,7 +39,8 @@ if err := shadow.Register(rt); err != nil { } ``` -Scheduled tasks and lifecycle hooks have no requester waiting. When starting the runtime, install the unified error sink, or these two kinds of errors are dropped: +Reminders and lifecycle hooks have no requester waiting. When starting the +Runtime, install the unified error sink, or these errors are dropped: ```go rt, err := gor.New( @@ -59,9 +65,15 @@ go run ./examples/shadow/cmd/shadow -cluster -addr 127.0.0.1:8082 -node-addr 127 go run ./examples/shadow/cmd/shadow -cluster -addr 127.0.0.1:8083 -node-addr 127.0.0.1:7373 -db ./data/cluster.db ``` -Run each in its own terminal. Every node serves the same HTTP API; send a request to any node and it is executed on the node that owns that entity, forwarded over the cluster transport when that node is a different one. The entity definitions and handlers are identical to the single-node service — clustering is a launcher concern, not a business-code one. +Run each in its own terminal. Every node serves the same HTTP API. A request +is executed on the Silo that owns its Grain. The Grain definitions and +handlers are the same as in the single-Silo service. -A node begins serving the moment it joins. As the nodes discover each other, which node owns which entity settles within about a second. During that window the same entity may be active on two nodes at once, so two writes to it can collide: one succeeds, the other fails and is returned to the caller — a client should retry it. This does not happen in the single-node service. The full boundary, including when this window opens beyond startup, is in [../../docs/programming-model.md](../../docs/programming-model.md). +A node begins serving when it joins. As Silos discover each other, Grain +ownership settles. During that window the same Grain may be active on two +Silos. One State write can then fail with a conflict. This does not happen in +the single-Silo service. The full boundary is in +[../../docs/programming-model.md](../../docs/programming-model.md). ## Calling it