diff --git a/docs-internal/registry-flatten-colocation-spec.md b/docs-internal/registry-flatten-colocation-spec.md index 4ec18091fb..3cd23d7c06 100644 --- a/docs-internal/registry-flatten-colocation-spec.md +++ b/docs-internal/registry-flatten-colocation-spec.md @@ -65,7 +65,7 @@ repo-root/ │ ├── claude/ codex/ opencode/ pi/ pi-cli/ # agent adapters — @agentos-software/*, JS only, no native/ │ │ ├── package.json agentos-package.json (kind: "agent") src/adapter.ts test/ │ │ -│ ├── vim/ vix/ # editors (vim = C in native/c/; vix = external binary, no source) +│ ├── vim/ # editor (C in native/c/) │ └── browserbase/ build-essential/ common/ everything/ # meta/external — manifest only, no native/ │ ├── toolchain/ ← shared WASM build infra (NOT a package; nothing here is @agentos-software/*) @@ -120,6 +120,15 @@ test-program fixtures stay in `toolchain/test-programs/` (not scattered into `kind: "agent"` in the manifest and have no `native/`. Command packages carry `kind: "software"` (or omit; default). This `kind` field is what distinguishes them — not folder location. +5. **`native/` holds the REAL upstream tool, patched — never a reimplementation.** + A command package's `native/` must build the genuine upstream program (GNU + coreutils, real `curl`/`git`/`jq`, GNU grep/sed/gawk/tar/gzip/diffutils, …) + fetched + pinned and patched for WASI — not a from-scratch Rust/C rewrite, a + stub, or a hand-rolled CLI over a library. The only exception is a tool whose + canonical upstream *is* the Rust project (`ripgrep`, `fd`). Several current + commands violate this (coreutils=uutils, grep, curl driver, and the + `secureexec-*` rewrites) — tracked in `registry-parity-worklist.md` + Cross-cutting #0; new packages must not add more. ## What moves where diff --git a/docs-internal/registry-parity-worklist.md b/docs-internal/registry-parity-worklist.md index 494d1220f7..61841d55a0 100644 --- a/docs-internal/registry-parity-worklist.md +++ b/docs-internal/registry-parity-worklist.md @@ -12,8 +12,30 @@ Status: worklist · Owner: registry · Last updated: 2026-07-07 > that only does `SELECT 1`. > > **Rules:** -> - **One jj rev per item.** New change per fix; `jj describe` with a clear -> conventional-commit message; do not batch unrelated fixes into one rev. +> - **🚧 REAL TOOL, NOT A REIMPLEMENTATION (the load-bearing rule).** Every command +> must be the **real upstream tool** (GNU coreutils, GNU grep/sed/gawk, real +> `curl`, real `git`, real `jq`, GNU tar/gzip/diffutils, …) compiled to +> `wasm32-wasip1` and **patched as needed** for WASI. Do **NOT** ship a +> from-scratch Rust/C rewrite, a stub, or a hand-rolled CLI over a library. +> Reimplementations drift from Linux behavior in a thousand small ways and are +> exactly why several commands fail parity. Sole exception: when the upstream +> canonical tool *is itself* the Rust project (**ripgrep**, **fd**) — then the +> real project is correct. Prefer the genuine upstream tool (real git, real +> grep) over a rewrite; a *popular, established* reimplementation is an +> acceptable fallback only when the real tool genuinely won't build. +> - **"Not possible" is a valid outcome — but only after trying really hard.** If a +> command cannot be built as the real (or an established) tool for WASI, do NOT +> hand-roll a custom replacement. Instead mark it **`NOT POSSIBLE (WASI)`** in +> this doc with a concrete explanation of exactly what blocks it (missing +> syscall, unsupported threading, sysroot gap, etc.) and what was tried. Exhaust +> real options first: patch the sysroot, patch the tool, stub the specific +> missing syscall — a genuine effort, not a quick bail. +> - **One jj rev per item.** Concretely: **`jj new` before starting each item**, +> make that command's fix *and* its e2e test in that single change, `jj describe` +> it with a clear conventional-commit message, then `jj new` again for the next +> item. One command per rev — never batch two commands (or unrelated changes) +> into one rev. Verify the folder + branch first (`pwd`, `jj log -r @`) since the +> working copy is shared. > - **Parity, not workarounds.** Fix the real cause (VFS syscall, shell semantics, > link conflict, missing feature). If a WASI limitation forces a deviation from > Linux, that is a finding to surface — not something to paper over in the test. @@ -32,6 +54,88 @@ Status: worklist · Owner: registry · Last updated: 2026-07-07 --- +## ⚠️ Cross-cutting #0 — Command provenance: replace reimplementations with real tools + +**This is the highest-leverage item and reshapes several below.** Audit revealed +that **most commands are NOT the real Linux tool** — they are custom Rust rewrites +(`secureexec-*` crates) or `uutils`, plus at least one hand-rolled C CLI (curl). +Per the load-bearing rule, each must become the **real upstream tool** compiled to +WASI and patched as needed. + +**Rule (your call):** an **established project** — whether it's the real upstream +tool *or* an established third-party package that does the real work (uutils, +jaq, etc.) — is **fine**. **Custom code we wrote ourselves** is **not** and must +be replaced with a real/established implementation. Audit of every command's +actual backing: + +### ✅ Established — keep (real upstream tool or established package doing the work) +| Command(s) | Backing | +|---|---| +| coreutils (`sh`+80) | **uutils** (`uucore`) — established Rust project | +| ripgrep (`rg`) | real ripgrep | +| duckdb, vim | real upstream C source, patched for WASI | +| sqlite3 **engine** | real SQLite amalgamation (⚠️ but the *CLI* is ours — see below) | +| jq | **jaq** (`jaq-core/std/json`) — established Rust jq | +| yq | jaq + `serde_yaml`/`toml`/`quick-xml` — established parsers (thin glue is ours) | +| sed | `sed` crate (published) | +| gawk (`awk`) | `awk-rs` crate (published) | +| tar | `tar` crate (established) | +| gzip | `flate2` (established) | +| diffutils (`diff`) | `similar` crate (established) | +| file | `infer` crate (established magic-byte lib; note: not real libmagic `file`) | + +### ❌ CUSTOM WE BUILT — flag & replace with a real/established impl +| Command | Status | What it actually is | Replace with | +|---|---|---|---| +| **curl** | TODO | our custom driver over a libcurl fork | real `curl` CLI (upstream `src/tool_*.c`) | +| **wget** | TODO | our 174-line `wget.c` | real GNU wget, or drop (curl covers it) | +| **http-get** | TODO | our 95-line `http_get.c` | drop, or a real tool | +| **git** | TODO | our hand-rolled git from `sha1`+`flate2` | **real git** (upstream C), patched for WASI — **NOT gitoxide** | +| **fd** | TODO | our `secureexec-fd` on raw `regex` (not sharkdp/fd) | real **fd** (sharkdp) | +| **findutils** (`find`,`xargs`) | TODO | our hand-rolled on `regex`/shims | real GNU findutils, or `uutils/findutils` | +| **tree** | TODO | our hand-rolled, zero deps | real `tree`, or an established one | +| **grep** | TODO | our `secureexec-grep` on raw `regex` (**not** an established grep pkg) | **real GNU grep**, or a popular established grep (ripgrep's `grep` crates) | +| **zip** | TODO | our 203-line `zip.c` over zlib/minizip (not Info-ZIP) | real Info-ZIP, or an established lib's CLI | +| **unzip** | TODO | our 669-line `unzip.c` over zlib/minizip | real Info-ZIP unzip | +| **sqlite3 CLI** | TODO | our 558-line `sqlite3_cli.c` (engine is real SQLite; the shell is ours) | real SQLite `shell.c` (its official CLI) | +| **vix** | DONE | from-scratch source-less drop-zone binary | deleted; real `vim` covers the editor slot | + +Note: `codex`/`codex-exec` = the rivet fork of OpenAI's codex — established fork, +external build (tracked separately in #9). + +**Objective:** replace each ❌ with a real/established implementation built to +`wasm32-wasip1` and patched only where WASI forces it. The ✅ rows stay. + +**Approach:** one command at a time, one jj rev each: swap our custom code for the +established source (fetched + pinned like sqlite/duckdb), wire into the toolchain, +patch for WASI, prove parity with real e2e tests. + +**Interaction with other items:** subsumes several below — curl (#6) is "build the +real curl," and the `no-test` packages (#12) that are ❌ here should move to a real +impl *before* their tests are written, so the tests validate real behavior. + +**Decisions (settled):** git → **real git** (not gitoxide). grep → **real GNU +grep**, or a popular established grep if the real one won't build. + +## Status tracking (how the driver reports progress in this doc) + +Update this doc as you go — it is the single source of truth for status. For each +❌ command, set one status and keep it current: + +- **`TODO`** — not started. +- **`IN PROGRESS`** — being built; note the current blocker if any. +- **`DONE`** — the real/established tool builds and passes a real un-skipped e2e + test; link the jj rev. +- **`NOT POSSIBLE (WASI)`** — only after a genuine effort. Write a concrete + explanation: exactly what blocks it, what you tried (sysroot patch, tool patch, + syscall stub), and why it can't be made to work. This is a documented dead-end, + never a silent fallback to a custom rewrite. + +Mark each row's status inline in the table (or as a short line under the command) +so a reader sees the whole board at a glance. + +--- + ## P0 — Runtime / VM correctness ### 1. brush-shell `>>` append truncates instead of appending @@ -82,14 +186,17 @@ Status: worklist · Owner: registry · Last updated: 2026-07-07 ## P1 — Broken shipped commands -### 6. curl — exits 1 on every operation (including `--version`) -- **Broken:** 24/30 `curl.test.ts` fail; every op returns exit 1, even - `curl --version`. The binary is non-functional in the VM as built. -- **Objective:** real curl behavior — GET/POST, headers (`-I`/`-D`), redirects - (`-L`), auth (`-u`), multipart (`-F`), file output (`-o`/`-O`), `-w`, `-K` — all - work like Linux curl over the runtime's socket/HTTP layer. **Not** a fetch-shim. +### 6. curl — reimplemented CLI, exits 1 on every operation (incl. `--version`) +- **Broken:** the `curl` command is a **hand-rolled `curl.c` driver** over a + libcurl fork, not the real curl command-line tool — so 24/30 `curl.test.ts` fail + and every op returns exit 1, even `curl --version`. +- **Objective (per Cross-cutting #0):** **build the real curl command-line tool** + (upstream `src/tool_*.c`) to `wasm32-wasip1` against the patched sysroot, + patched only where WASI forces it — replacing the custom driver. All real curl + behavior (GET/POST, `-I`/`-D`, `-L`, `-u`, `-F`, `-o`/`-O`, `-w`, `-K`) then + works because it *is* curl, not a shim. - **Proof:** `software/curl/test/` (the existing 30 tests) pass un-weakened. -- **rev:** `fix(curl): make curl functional in the VM (real HTTP over runtime sockets)` +- **rev:** `fix(curl): build the real curl CLI for WASI; drop the custom driver` ### 7. zip / unzip — hostile-archive hardening cases fail (3 each) - **Broken:** fallback parser doesn't reject a wrapping local offset, doesn't skip @@ -123,36 +230,15 @@ Status: worklist · Owner: registry · Last updated: 2026-07-07 - **Proof:** codex builds in CI/dev; `software/codex-cli/test/` runs un-skipped. - **rev:** `build(codex-cli): make the codex-rs fork build reproducible` -### 10. vix — external binary, no source or build recipe in repo -- **Broken:** `EXTERNAL_COMMANDS` — a hand-built WASM binary dropped in with **no - source and no build pipeline** committed. Also the shipped - `software/vix/dist/package.aospkg` is a **1100-byte manifest stub that does NOT - embed the wasm** — the real 214 KB binary is injected at build time. -- **Provenance (recovered):** vix is a **from-scratch 309-line single-file C - editor** (`vix.c`) — *not* a port of vis/neatvi/nvi/busybox/elvis. It avoids - `termios.h` entirely, using three guest imports - (`host_tty::{isatty,get_size,set_raw_mode}`, same ABI as - `toolchain/.../c/programs/pty_probe.c`) for raw mode + window size, plus VT100; - supports insert mode and `:w`/`:wq`/`:q`. Built by Claude thread - `46371327-9e48-4c4c-8150-10dd21d7bf0f` (2026-06-29). Source, recipe, and binary - are **preserved and reproducible** in - `~/progress/agent-os/2026-06-28-just-shell-fix/` → `vix.c`, `BUILD-vix.md`, - `vix.wasm` (214 KB, md5 `a6e650f03493ad0dff230691d67ee3bd`). Original build: - ``` - $WASI_SDK/bin/clang --target=wasm32-wasip1 --sysroot=$WASI_SDK/share/wasi-sysroot \ - -O2 -I /include -o vix.wasm vix.c # vanilla sysroot, no termios - ``` -- **Decide first:** keep vix at all? A real **vim** (11.5 MB, `wasm32-wasip1`) now - exists (#11), so vix's role is "tiny editor with no bundled runtime." Either (a) - keep it and properly source it, or (b) drop the package in favor of vim. -- **Objective (if kept):** commit `vix.c` into `software/vix/native/c/`, add a real - build recipe (host_tty imports + the sysroot), wire it into `toolchain` as a - normal C command, **remove the `EXTERNAL_COMMANDS` drop-zone hack**, and fix the - `.aospkg` so it actually embeds the binary. Then a real editor e2e test (raw-mode - keystrokes through the VM PTY → `:wq` writes the VFS file). -- **Proof:** `make -C toolchain cmd/vix` builds from committed source (no dropped - binary); `software/vix/test/` drives real modal editing and asserts the written file. -- **rev:** `build(vix): commit vix.c + build recipe; drop EXTERNAL_COMMANDS; embed binary in aospkg` +### 10. vix — DONE (deleted) +- **Resolved:** `vix` was a from-scratch, source-less drop-zone binary — exactly + the kind of hand-rolled artifact this repo should not carry. **Removed entirely** + (package dir, shell import/dep, `EXTERNAL_COMMANDS` Makefile hack, README rows, + website registry entry) in rev + `chore(registry): remove vix package; document real-tool (no-reimplementation) principle`. + Real `vim` (#11) covers the editor slot. Preserved source (`vix.c`, `BUILD-vix.md`, + `vix.wasm`) remains in `~/progress/agent-os/2026-06-28-just-shell-fix/` if ever + needed. No further work. --- diff --git a/packages/shell/package.json b/packages/shell/package.json index f76cda5d3c..254004e6b4 100644 --- a/packages/shell/package.json +++ b/packages/shell/package.json @@ -35,7 +35,6 @@ "@agentos-software/tree": "workspace:*", "@agentos-software/unzip": "workspace:*", "@agentos-software/vim": "workspace:*", - "@agentos-software/vix": "workspace:*", "@agentos-software/wget": "workspace:*", "@agentos-software/yq": "workspace:*", "@agentos-software/zip": "workspace:*", diff --git a/packages/shell/src/main.ts b/packages/shell/src/main.ts index 878c1cf539..f438278d19 100644 --- a/packages/shell/src/main.ts +++ b/packages/shell/src/main.ts @@ -41,7 +41,6 @@ import ripgrep from "@agentos-software/ripgrep"; import sed from "@agentos-software/sed"; import sqlite3 from "@agentos-software/sqlite3"; import vim from "@agentos-software/vim"; -import vix from "@agentos-software/vix"; import tar from "@agentos-software/tar"; import tree from "@agentos-software/tree"; import unzip from "@agentos-software/unzip"; @@ -159,12 +158,12 @@ const software: SoftwareInput[] = [ .map(withLocalCommandFallback) .filter((input): input is SoftwareInput => input !== null); -// The vi-like editors ship as software packages: vix, and vim (which carries -// its runtime tree + VIMRUNTIME via the manifest `provides`). An unbuilt -// package is a valid empty placeholder — skip it rather than projecting a -// command-less package (build them locally: drop the wasm binaries into -// toolchain/target/.../commands and run `just software-build`). -for (const editor of [vix, vim] as RegistryPackage[]) { +// vim ships as a software package (which carries its runtime tree + VIMRUNTIME +// via the manifest `provides`). An unbuilt package is a valid empty placeholder +// — skip it rather than projecting a command-less package (build it locally: +// drop the wasm binary into toolchain/target/.../commands and run +// `just software-build`). +for (const editor of [vim] as RegistryPackage[]) { if (isUsablePackageFile(editor.packagePath)) { software.push({ packagePath: editor.packagePath }); continue; diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 62e8dbead3..38e3dc4dc6 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -2877,9 +2877,6 @@ importers: '@agentos-software/vim': specifier: workspace:* version: link:../../software/vim - '@agentos-software/vix': - specifier: workspace:* - version: link:../../software/vix '@agentos-software/wget': specifier: workspace:* version: link:../../software/wget @@ -3741,27 +3738,6 @@ importers: specifier: ^2.1.9 version: 2.1.9(@types/node@22.19.15) - software/vix: - devDependencies: - '@agentos-software/manifest': - specifier: workspace:* - version: link:../../packages/manifest - '@agentos/test-harness': - specifier: workspace:* - version: link:../../test-harness - '@rivet-dev/agentos-toolchain': - specifier: workspace:* - version: link:../../packages/agentos-toolchain - '@types/node': - specifier: ^22.10.2 - version: 22.19.15 - typescript: - specifier: ^5.9.2 - version: 5.9.3 - vitest: - specifier: ^2.1.9 - version: 2.1.9(@types/node@22.19.15) - software/wget: devDependencies: '@agentos-software/manifest': diff --git a/software/README.md b/software/README.md index 71780e40c7..7f15771bb8 100644 --- a/software/README.md +++ b/software/README.md @@ -71,7 +71,6 @@ per-binary entry point; it dispatches to whichever toolchain owns the command: | C | `zip unzip envsubst sqlite3 curl wget duckdb` | `make -C c sysroot build/` + per-command install | | codex | `codex`, `codex-exec` | the codex fork build (needs the fork checkout) | | C | `vim` (pinned upstream clone + bridge in `c/vim/`) | `make -C c sysroot build/vim` + install | -| external | `vix` | validates the hand-built binary is in the drop zone; errors with instructions otherwise | The default native build (`toolchain`) compiles the fast command gate to `wasm32-wasip1` with a patched std (`-Z build-std`, `toolchain/std-patches/`), runs @@ -98,8 +97,6 @@ Exceptions: vim tag and compiles it against the patched sysroot + the termios/termcap bridge in `software/vim/native/c/vim-bridge/` (its runtime tree is staged by the package `scripts/stage-runtime.mjs` and applied via manifest `provides`). -- `vix` is the one remaining external drop-zone binary (no source pipeline): - place the hand-built wasm at `toolchain/target/.../commands/vix`. ## Publishing diff --git a/software/vix/agentos-package.json b/software/vix/agentos-package.json deleted file mode 100644 index 5cc6db8a35..0000000000 --- a/software/vix/agentos-package.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "name": "vix", - "commands": [ - "vix" - ], - "registry": { - "title": "vix", - "description": "Lightweight vi-style text editor.", - "category": "editors" - } -} diff --git a/software/vix/package.json b/software/vix/package.json deleted file mode 100644 index 04b98e8d28..0000000000 --- a/software/vix/package.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "name": "@agentos-software/vix", - "version": "0.1.0", - "type": "module", - "license": "Apache-2.0", - "description": "vix editor for secure-exec VMs (wasm build)", - "main": "./dist/index.js", - "types": "./dist/index.d.ts", - "files": [ - "dist", - "!dist/package", - "!dist/package.tar" - ], - "exports": { - ".": { - "types": "./dist/index.d.ts", - "import": "./dist/index.js", - "default": "./dist/index.js" - } - }, - "scripts": { - "build": "agentos-toolchain stage --commands-dir ../../toolchain/target/wasm32-wasip1/release/commands --if-missing skip && tsc && agentos-toolchain build", - "check-types": "tsc --noEmit", - "test": "vitest run test/ --passWithNoTests" - }, - "devDependencies": { - "@agentos-software/manifest": "workspace:*", - "@rivet-dev/agentos-toolchain": "workspace:*", - "@types/node": "^22.10.2", - "typescript": "^5.9.2", - "@agentos/test-harness": "workspace:*", - "vitest": "^2.1.9" - } -} diff --git a/software/vix/src/index.ts b/software/vix/src/index.ts deleted file mode 100644 index dba100e2e1..0000000000 --- a/software/vix/src/index.ts +++ /dev/null @@ -1,5 +0,0 @@ -import type { SoftwarePackageRef } from "@agentos-software/manifest"; - -const packagePath = new URL("./package.aospkg", import.meta.url).pathname; - -export default { packagePath } satisfies SoftwarePackageRef; diff --git a/software/vix/test/manifest.test.ts b/software/vix/test/manifest.test.ts deleted file mode 100644 index 89139d33b8..0000000000 --- a/software/vix/test/manifest.test.ts +++ /dev/null @@ -1,19 +0,0 @@ -import { readFileSync } from "node:fs"; -import { join } from "node:path"; -import { describe, expect, it } from "vitest"; - -const packageDir = new URL("..", import.meta.url).pathname; - -describe("package manifest", () => { - it("declares command binaries", () => { - const manifest = JSON.parse( - readFileSync(join(packageDir, "agentos-package.json"), "utf8"), - ); - - expect(manifest.commands?.length ?? 0).toBeGreaterThan(0); - for (const command of manifest.commands) { - expect(typeof command).toBe("string"); - expect(command.length).toBeGreaterThan(0); - } - }); -}); diff --git a/software/vix/tsconfig.json b/software/vix/tsconfig.json deleted file mode 100644 index 03ce790ab7..0000000000 --- a/software/vix/tsconfig.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "extends": "../tsconfig.base.json", - "compilerOptions": { - "outDir": "./dist", - "rootDir": "./src" - }, - "include": ["src/**/*"] -} diff --git a/toolchain/Makefile b/toolchain/Makefile index 03b6c77496..e14e52b982 100644 --- a/toolchain/Makefile +++ b/toolchain/Makefile @@ -208,11 +208,9 @@ wasm: vendor patch-vendor patch-std wasm-opt-check # (cargo package cmd-, via wasm-cmd) # C the C opt-in set below (toolchain/c, patched sysroot) # codex the codex fork build (codex, codex-exec) -# external hand-built drop-zone binaries with no source pipeline yet # `just toolchain-cmd ` calls this. `make wasm` builds the fast # Rust set; `make -C c programs install` builds/installs the fast C set. C_COMMANDS := zip unzip envsubst sqlite3 curl wget duckdb vim http_get -EXTERNAL_COMMANDS := vix .PHONY: cmd/% cmd/%: @@ -225,15 +223,8 @@ cmd/%: $(MAKE) -C c sysroot "build/$$src" && $(MAKE) -C c install COMMANDS="$$name"; \ elif [ "$$name" = "codex" ] || [ "$$name" = "codex-exec" ]; then \ $(MAKE) codex-required; \ - elif echo " $(EXTERNAL_COMMANDS) " | grep -q " $$name "; then \ - if [ -f "$(COMMANDS_DIR)/$$name" ]; then \ - echo "cmd/$$name: external hand-built binary present at $(COMMANDS_DIR)/$$name"; \ - else \ - echo "ERROR: '$$name' has no source pipeline yet — place the hand-built wasm binary at $(COMMANDS_DIR)/$$name (see software/README.md)"; \ - exit 1; \ - fi; \ else \ - echo "ERROR: unknown command '$$name' (no software/*/native/crates/cmd-$$name or crates/commands/$$name; not in C_COMMANDS, codex, or EXTERNAL_COMMANDS)"; \ + echo "ERROR: unknown command '$$name' (no software/*/native/crates/cmd-$$name or crates/commands/$$name; not in C_COMMANDS or codex)"; \ exit 1; \ fi diff --git a/website/src/generated/registry.json b/website/src/generated/registry.json index b52bbb0024..d65f37e211 100644 --- a/website/src/generated/registry.json +++ b/website/src/generated/registry.json @@ -415,18 +415,6 @@ "priority": 0, "package": "@agentos-software/http-get", "status": "available" - }, - { - "slug": "vix", - "title": "vix", - "description": "Lightweight vi-style text editor.", - "types": [ - "software" - ], - "category": "editors", - "priority": 0, - "package": "@agentos-software/vix", - "status": "available" } ] }