Skip to content

refactor: move experiment workers out of the node runtime #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BernardLowe merged 1 commit into from
Oct 12, 2026
Merged

refactor: move experiment workers out of the node runtime #73

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 6 additions & 12 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@
This repository owns the user-controlled MALT local data runtime: CLI, local
daemon, accepted-root policy, UnixFS application semantics, Merkle DAG import
compatibility, pluggable transport capabilities, and payload-byte verification.
It also keeps the runtime-owned executable adapters needed to measure those exact behaviors;
the evaluator that plans and interprets campaigns lives in `malt-evaluation`.
This is the edge node implementation. Reproducible measurement drivers, replay
workers, experiment protocols, and accounting policy live in `malt-evaluation`
and consume public node/runtime capabilities.

## Boundaries

Expand All @@ -28,9 +29,9 @@ the evaluator that plans and interprets campaigns lives in `malt-evaluation`.
- Keep IPFS-style Merkle DAG import and compatibility policy here. A Merkle DAG
root is a compatibility output, not a MALT-authenticated root or ProofList.
- Keep benchmark suites, run plans, comparison policy, result schemas, and
result provenance in `malt-evaluation`. Code under `tools/evaluation` is a
pinned process adapter, not a supported runtime command or an evaluator
source of truth.
result provenance in `malt-evaluation`. Do not place experiment workers or
experiment-specific accounting in this node repository. Expose only reusable
node/runtime capabilities and storage observation needed by external callers.

## Package Ownership

Expand Down Expand Up @@ -163,13 +164,6 @@ the evaluator that plans and interprets campaigns lives in `malt-evaluation`.
select configured runtime services, call `application/` use cases, and
format results, not become a second UnixFS, trust, or plan-composition
implementation. `cmd/` contains only the supported `malt` product binary.
- `tools/evaluation/cmd/` owns evaluator-launched runtime process adapters.
Preserve their external binary names and wire contracts, but do not treat
their Go packages as a compatibility surface.
- `internal/evaluation/` owns shared implementation for those adapters,
including disposable-Gateway instance authentication, bootstrap, raw-CAS,
and selective-CAR transport. Production packages and `cmd/malt` must not
import it.
- Preserve `internal/architecture` dependency tests when adding packages or
moving responsibilities.

Expand Down
19 changes: 8 additions & 11 deletions ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,11 @@ It owns:
- application-level payload-byte verification.
- runtime-side encrypted snapshot creation, restore, and automatic scheduling.

Runtime-specific benchmark process adapters also live in this repository under
`tools/evaluation`. They exercise this implementation at a pinned commit, but
do not own benchmark plans, suites, comparison policy, result schemas, or
result provenance; those belong in `malt-evaluation`.
Benchmark process adapters, replay workers, experiment fixtures, accounting,
plans, suites and provenance belong in `malt-evaluation`. Its adapters exercise
this node through public capabilities at a pinned module version. This runtime
and Gateway provide edge and cloud node implementations; measurements remain
outside the node implementation.

It depends on `github.com/dewebprotocol/malt-core` for canonical graph types,
typed query/candidate/batch contracts, local verification, CID rules, and
Expand Down Expand Up @@ -124,7 +125,7 @@ merkledag -> CID/link replay + fixed profile transport
trust -> accepted/candidate root persistence
transport -> Gateway/local/hybrid adapters; never imports unixfs, merkledag, or trust
bucketsync -> nodeapi.DatasetBranch + independent durable synchronization metadata
tools/evaluation/cmd -> internal/evaluation + public runtime capabilities
malt-evaluation adapters -> public nodeapi / runtime capabilities (external dependency)
```

`transport.Client` is one reusable HTTP connection, but consumers depend on
Expand All @@ -134,8 +135,8 @@ the fixed `PostMerkleDAGResolve` and `PostMerkleDAGRead` capabilities; there is
no arbitrary profile-route escape hatch. Transport results remain untrusted.
The public transport also does not expose evaluation instance credentials,
bootstrap control, unchecked raw-CAS reads, or the selective-CAR route. Those
disposable-Gateway capabilities live under `internal/evaluation` and can only
be composed by evaluator adapters.
disposable-Gateway capabilities live under `malt-evaluation/internal/nodeeval`
and can only be composed by evaluator adapters.

`transport/local` is a bounded, atomic, owner-private durable CAS and verifies
the complete body on both `Get` and `Has`. `transport/hybrid` is transport-level
Expand Down Expand Up @@ -287,8 +288,6 @@ profile; it does not change MALT Core or the existing plaintext semantics of
## Packages

- `cmd/malt`: CLI and daemon process lifecycle.
- `tools/evaluation/cmd`: evaluator-launched process adapters. Their executable
names and wire contracts are campaign inputs, not supported user CLI.
- `application`: reusable trusted-root, UnixFS, and Merkle DAG use cases shared
by command and daemon adapters.
- `application/backup`: encrypted MALT-native filesystem, local fingerprint, Bucket
Expand Down Expand Up @@ -394,8 +393,6 @@ profile; it does not change MALT Core or the existing plaintext semantics of
- `internal/securefile`: owner-only Unix file modes and protected
owner-and-SYSTEM Windows file DACLs.
- `internal/cas`: runtime-side CAS helpers and byte verification.
- `internal/evaluation`: private adapter support, including evaluation Gateway
authentication/control transport and shared workload machinery.
- `unixfs/model`: UnixFS application values and path rules.
- `unixfs`: verified UnixFS reader/writer facade, staging, materialization,
and payload verification.
Expand Down
28 changes: 13 additions & 15 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -532,20 +532,17 @@ and cannot fall back to the public CAS namespace.

## Evaluation adapters

`cmd/` contains only the supported `malt` product binary. Runtime-specific
benchmark process adapters live under `tools/evaluation/cmd`, with shared
private support under `internal/evaluation`. They preserve their external
`malt-eval-*` executable and wire contracts, but are not a public Go API or
supported user CLI.

Evaluation instance authentication, bootstrap control, unchecked raw-CAS
fetches, and selective-CAR transport are deliberately confined to that private
boundary. The public `transport` package cannot invoke them. Benchmark suites,
campaign plans, comparison policy, result schemas, and provenance remain owned
by the separate
[malt-evaluation](https://github.com/DeWebProtocol/malt-evaluation) repository.
See [tools/evaluation/README.md](./tools/evaluation/README.md) for the adapter
inventory and build boundary.
`cmd/` contains only the supported `malt` product binary. Replay workers,
benchmark drivers, experimental transports, fixtures, accounting and result
schemas live in the separate
[malt-evaluation](https://github.com/DeWebProtocol/malt-evaluation) repository,
under `adapters/node/cmd` and `internal/nodeeval`.

The evaluator calls the public `nodeapi`, UnixFS, Merkle DAG and transport
capabilities. The node provides storage and authentication behavior; the
evaluator defines workloads and measurement boundaries. Gateway and this
runtime are cloud and edge implementations of node capabilities, respectively.
Evaluation builds pin the node module separately from the evaluator revision.

The transport exposes bounded ordered CAS `PutBatch`/`HasBatch` and a
typed diagnostic metrics snapshot. Merkle DAG import and verified link replay
Expand Down Expand Up @@ -627,7 +624,8 @@ for completion. None of these operations accepts a candidate automatically.
The candidate API is a complete-view workflow, not a stateless transition proof.

Evaluation tooling can export bounded file snapshots with
`go run ./tools/evaluation/cmd/malt-eval-rooted-file-trace --input snapshots.json`.
`go run ./adapters/node/cmd/malt-eval-rooted-file-trace --input snapshots.json`
from a `malt-evaluation` checkout.
The input is an array of complete versions, each containing `files` (canonical
relative paths mapped to base64 bytes) and `queries` (canonical paths).
`--backend ipa` and `--chunk-size` select the commitment and file chunk geometry.
Expand Down
6 changes: 3 additions & 3 deletions cmd/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@ This namespace contains only the supported `malt` runtime binary:
go build -o bin/malt ./cmd/malt
```

Benchmark workers are not product commands. They live under
[`tools/evaluation/cmd`](../tools/evaluation/cmd) so packaging and API reviews
cannot confuse evaluator process adapters with the user-facing CLI.
Benchmark workers are not product commands. They live in
[`malt-evaluation/adapters/node/cmd`](https://github.com/DeWebProtocol/malt-evaluation/tree/main/adapters/node/cmd)
and call public node/runtime capabilities.
41 changes: 0 additions & 41 deletions docs/evaluation-local-files.md
View file
Open in desktop

This file was deleted.

4 changes: 2 additions & 2 deletions docs/go-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,8 +185,8 @@ defines no network codec, discovery rule, or wire identifier.

The public transport does not expose evaluation instance tokens, bootstrap
control, unchecked raw-CAS reads, or the selective-CAR route. Those
capabilities are private to the pinned process adapters under
`tools/evaluation`; they are not a supported integration surface.
capabilities are private to the process adapters in the separate
`malt-evaluation` repository; they are not a supported node integration surface.

`Metrics` returns inexpensive monotonic counters. `MetricsWithStorage` also
requests Gateway's O(live KV entries) logical scan and should be used only by
Expand Down
2 changes: 1 addition & 1 deletion docs/runtime-migration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ not a current build contract. The original plan is available in
| `internal/runtime` | Reusable composition of local services and transports |
| `cmd/malt` | CLI and daemon entrypoints |
| `merkledag` | Isolated IPFS-compatible import and CID/link replay |
| `tools/evaluation` | Private measured workers, separate from production policy |
| Evaluation workers | Moved to `malt-evaluation/adapters/node/cmd`; consume public node/runtime capabilities |

The runtime's Go module namespace remains `github.com/dewebprotocol/malt-client`
until its separately governed module migration. The module pins published Core
Expand Down
2 changes: 1 addition & 1 deletion docs/v0.0.5-parity.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ repository at v0.0.5.
| ArcTable, KV, and CAS persistence | gateway | migrated; not imported by runtime or core |
| standalone CAS process | none | deliberately removed; gateway owns product CAS |
| evaluator and benchmark suites | [malt-evaluation](https://github.com/DeWebProtocol/malt-evaluation) | former v0.0.5 evaluator retired from the active tree and recoverable from the [immutable standalone import commit](https://github.com/DeWebProtocol/malt-evaluation/commit/774854b5d90c1ba5d57c3d74619dd97a78fd9dcf) and Git history; active workloads live at the repository root |
| current runtime measurement adapters | malt | evaluator-launched `malt-eval-*` process boundaries live under `tools/evaluation/cmd`; shared implementation is private under `internal/evaluation`, while plans, suites, schemas, and provenance remain in `malt-evaluation` |
| current runtime measurement adapters | malt-evaluation | `adapters/node/cmd` and `internal/nodeeval` own workers and experimental support; the runtime exports only public node/application capabilities |
| Merkle DAG gateway compatibility reads | gateway + malt | gateway traverses UnixFS and returns CID-bound blocks; the runtime independently hashes and replays them under a distinct non-MALT profile |
| profile-specific transport escape hatch | none | arbitrary route/body transport was removed; public Merkle DAG compatibility uses fixed resolve/read capabilities, while evaluation-only bootstrap/raw-CAS/selective-CAR routes are private adapter implementation |
| generic IPLD `ParseBlock`/link helpers | malt | retired: unused generic parser removed; supported import and verified DAG-PB/raw/DAG-CBOR/DAG-JSON replay remain in `merkledag` |
Expand Down
4 changes: 0 additions & 4 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,7 @@ require (
github.com/ipld/go-ipld-prime v0.24.0
github.com/multiformats/go-multibase v0.3.0
github.com/multiformats/go-multihash v0.2.3
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
github.com/spf13/cobra v1.10.2
github.com/syndtr/goleveldb v1.0.0
golang.org/x/crypto v0.57.0
golang.org/x/sys v0.48.0
golang.org/x/term v0.46.0
Expand All @@ -33,7 +31,6 @@ require (
github.com/gammazero/deque v1.2.1 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
Expand Down Expand Up @@ -71,7 +68,6 @@ require (
go.uber.org/zap v1.28.0 // indirect
golang.org/x/exp v0.0.0-20260603202125-055de637280b // indirect
golang.org/x/sync v0.23.0 // indirect
golang.org/x/text v0.42.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/protobuf v1.36.11 // indirect
lukechampine.com/blake3 v1.4.1 // indirect
Expand Down
Loading
Loading