diff --git a/README.md b/README.md index faa92e8..57dd8b2 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ see [Antigravity 2.0 (IDE) sessions](#antigravity-20-ide-sessions). By default, the daemon binds a different ephemeral port each `agy` session. agy-reader discovers the live CLI endpoint and its CSRF token from local -processes on Linux, with session logs as a fallback for older versions. +processes on Linux and macOS, with `cli.log` as a fallback for the HTTP port. The explicit `export-offline` command also reads supported SQLite sessions directly, without a running agy daemon or proxy. See [Offline export](#offline-export). @@ -185,6 +185,10 @@ point of the contract. ### Sidecar contract +Watch can also preserve a first transcript whose format the reader cannot yet +verify. Its `agyReader.completeness` contains `"state": "unverified"` and a +reason. Doctor counts these files even after agy removes the source. + For every syncable `~/.gemini/antigravity-cli/conversations/.db` or `.pb`, agy-reader writes `.trajectory.json` in the same directory. When both `.db` and `.pb` exist, as agy 1.2.17 leaves them for a forked @@ -404,17 +408,18 @@ someone deliberately forging cache files or filesystem metadata. A changed activ session still needs content verification and, after a successful sync, parent-link resolution across the sidecar corpus. -On Linux, CLI discovery checks the daemon's open session log and owned listening -socket before using credentials. By default, discovery uses the -`ANTIGRAVITY_LS_ADDRESS` and `ANTIGRAVITY_CSRF_TOKEN` values agy exports to a -live tool process. A process environment is readable only by its own user, so -the token stays private to you. A running watcher keeps that token in memory -after the tool exits. A fresh watcher that you start yourself, such as the -systemd service, cannot authenticate until it catches another tool process; -commands shorter than its polling interval may be missed entirely. +On Linux and macOS, run `agy-reader --watch` under the same account as a normal +`agy` session. The reader checks the daemon's open session log and listening +socket, then uses the credentials agy exports to a readable tool process. On +macOS, this means a subprocess started by that `agy`, such as a shell command. +The watcher keeps the token in memory after the tool exits, so later text-only +turns can sync. A new watcher or `doctor` needs to see a live tool process to +discover the token for a plain launch. Commands shorter than the watch interval +can finish between polls. The macOS CLI path was verified with agy 1.5.10 on +Apple Silicon. -**Single-user machines only:** for reliable discovery even while agy is idle or -the watcher restarts, launch agy with a fresh token: +**Single-user Linux machines only:** for reliable discovery even while agy is +idle or the watcher restarts, launch agy with a fresh token: ```sh agy --csrf_token="$(uuidgen)" @@ -445,10 +450,11 @@ your user and root can read, and discovery reads it there. So a watcher you start yourself, such as the systemd service, can authenticate to that agy even while it is idle. -Every discovery path requires readable `/proc` entries in the same Linux network -namespace and a daemon log under the selected root (`cli.log` or `log/cli-*.log`). -This avoids stale `cli.log` ports left by short-lived CLI invocations. Other -platforms, restricted `/proc` access, and custom log locations can use +CLI process discovery requires an open daemon log under the selected root +(`cli.log` or `log/cli-*.log`). Linux uses `/proc` in the same network namespace. +macOS uses the system `/usr/sbin/lsof` to check the daemon's files and sockets, +and the OS process table to find readable tools under the same user. Restricted +process access, custom log locations, and other platforms can use the `ANTIGRAVITY_DAEMON_URL` and `ANTIGRAVITY_CSRF_TOKEN` overrides. Authentication rejection stops the batch with one `authentication failed; sync blocked` message per poll instead of @@ -686,6 +692,8 @@ It checks the following: rejected until a token is found or pinned via `ANTIGRAVITY_CSRF_TOKEN`. - **auth** — a reachable CLI daemon that rejects a read-only RPC is reported as `rejected — sync blocked`, and fails doctor even if the IDE root is healthy. + With a plain CLI launch, run doctor while agy is running a readable external + tool so doctor can discover the current token itself. - **agy version** — the running `agy --version` compared against the baseline recorded in `COMPATIBILITY.md`. A skew means the format audit should re-run. - **sidecars** — SQLite and legacy encrypted `.pb` sessions are verified, @@ -741,7 +749,13 @@ matches, watch leaves the sidecar unchanged and retries later. One-shot reads may return an existing cache with a warning; that fallback cannot refresh its contents or freshness evidence. -Unknown source fields, schemas, enum values, or projections fail closed and +When the persisted identity is available and the reader encounters unknown +protobuf fields or an unsupported display projection, watch saves the identified +presentation if the sidecar path is empty. An existing file always wins over a +provisional capture. After a compatibility update, watch replaces the provisional +file on successful verification if the source and daemon are still available. + +Unknown source fields, schemas, enum values, or projections fail verification and need a compatibility update. When the source itself causes the refusal (it cannot be read, lies outside the supported profile, or has a projection the profile does not support), watch logs it once and does not fetch that @@ -760,8 +774,8 @@ watcher or repair historical sidecars until the updated reader is run. **`Auto-discovery failed and ANTIGRAVITY_DAEMON_URL is not set`** -On Linux, `agy-reader` first looks for the active CLI process and its launch -token or exported tool credentials. As a fallback, it scans `cli.log` inside the +On Linux and macOS, `agy-reader` first looks for the active CLI process and its +credentials. As a fallback, it scans `cli.log` inside the session root directory (`~/.gemini/antigravity-cli/` or `$ANTIGRAVITY_CLI_ROOT`) to locate the active HTTP port. For an IDE root (`~/.gemini/antigravity`) it scans @@ -776,7 +790,8 @@ will fail. You can troubleshoot by: 1. Manually overriding the port if necessary. Find the port using: ```bash ss -tlnp 2>/dev/null | grep agy # Linux - lsof -iTCP -sTCP:LISTEN -anP | grep agy # macOS + pgrep -x agy # macOS: find the CLI PID + lsof -nPa -p -iTCP -sTCP:LISTEN # macOS: inspect that PID's listeners ``` Use the port explicitly logged as `for HTTP`, not the HTTPS/gRPC port. For CLI versions that require CSRF, also supply the matching token exported @@ -785,6 +800,15 @@ will fail. You can troubleshoot by: export ANTIGRAVITY_DAEMON_URL=http://127.0.0.1: ``` +**`authentication failed; sync blocked` on macOS** + +Start the watcher under the same macOS account as `agy`. If the daemon is idle, +wait for agy to run a shell command or another external tool. A shorter interval +such as `--watch-interval=2s` helps catch tools that finish quickly. If you set +an endpoint or token override for a previous session, unset it and restart the +watcher. Use `agy-reader doctor --root "$HOME/.gemini/antigravity-cli"` while the +tool is running to check authentication and sidecar coverage. + **`connection refused`** The daemon only listens while `agy` (Antigravity CLI) is running, and the port @@ -813,7 +837,7 @@ default locations `~/.gemini/antigravity-cli` / `~/.gemini/antigravity`. | ------------------------ | ------------------------------------------------------------- | ----------------------------------- | | `ANTIGRAVITY_DAEMON_URL` | Daemon base URL override (optional, auto-detected by default) | unset (optional fallback) | | `ANTIGRAVITY_CLI_ROOT` | Pin a single session root (suppresses store discovery) | unset (each default store that exists) | -| `ANTIGRAVITY_CSRF_TOKEN` | CSRF token override for daemons that enforce one | unset (auto-discovered for Linux CLI and the IDE) | +| `ANTIGRAVITY_CSRF_TOKEN` | CSRF token override for daemons that enforce one | unset (auto-discovered for Linux and macOS CLI and the IDE) | | `AGY_READER_LIVE` | Enable live daemon smoke test | unset (test skips) | | `AGY_READER_TEST_UUID` | Cascade id to use in the live test | unset | @@ -851,7 +875,9 @@ Do not combine it with the event-driven service configuration. ### launchd (macOS) -`~/Library/LaunchAgents/dev.mjacobs.agy-reader.plist`: +Save the following as `~/Library/LaunchAgents/dev.mjacobs.agy-reader.plist`. +Use the logged-in user's LaunchAgent so the watcher can see that account's +`agy` processes. Replace both example paths with absolute paths on your Mac. ```xml @@ -863,24 +889,30 @@ Do not combine it with the event-driven service configuration. dev.mjacobs.agy-reader ProgramArguments - /usr/local/bin/agy-reader + /path/to/agy-reader --watch + --root + /path/to/home/.gemini/antigravity-cli RunAtLoad KeepAlive - StandardOutPath - /tmp/agy-reader.out - StandardErrorPath - /tmp/agy-reader.err ``` -Load with -`launchctl load -w ~/Library/LaunchAgents/dev.mjacobs.agy-reader.plist`. -Replace `/usr/local/bin/agy-reader` with your installed binary's absolute path. +Load and inspect the agent from your macOS login session: + +```sh +launchctl bootstrap "gui/$(id -u)" "$HOME/Library/LaunchAgents/dev.mjacobs.agy-reader.plist" +launchctl print "gui/$(id -u)/dev.mjacobs.agy-reader" +``` + +Start `agy` as usual. The agent waits for the daemon and can sync once it sees +an external tool with readable credentials. Run only one watcher for the store. +To switch to a foreground watcher for troubleshooting, first unload the agent +with `launchctl bootout "gui/$(id -u)/dev.mjacobs.agy-reader"`. ## Testing @@ -975,7 +1007,7 @@ symlinks it into the gitignored agent skill directories (`.claude/skills/`, Active development. Currently supports: -- CLI port and CSRF discovery from Linux processes, with `cli.log` fallback; +- CLI port and CSRF discovery from Linux and macOS processes, with `cli.log` fallback; IDE discovery from `language_server.log` and `main.log`. - Antigravity 2.0 (IDE) conversations, discovered automatically alongside the CLI store, including CSRF token discovery for the IDE daemon. diff --git a/docs/compatibility/agy-1.5.10.md b/docs/compatibility/agy-1.5.10.md index d423da1..deeb14f 100644 --- a/docs/compatibility/agy-1.5.10.md +++ b/docs/compatibility/agy-1.5.10.md @@ -75,3 +75,14 @@ linked to the pinned compression format. Fedora's system Python uses zlib-ng, which fails the same earlier overlay check on the baseline commit. The separate agy-pair suite passes all 64 tests on system Python, including the pidfd cases skipped by the schema verification interpreter. + +## Policy guardian configuration in a later test turn + +A follow-up turn advanced the watcher sidecar from 317 to 319 steps. SQLite +and the daemon reached 320, including the completed reply. Three unknown +`PolicyGuardianConfig` fields in the new executor metadata prevented the +final sidecar write. The reader now decodes `deployment` (15), +`enable_irreversible_vetting` (18), and +`enable_reversibility_adjudication` (19) from the authenticated snapshot. +The new profile verifies all 320 steps against both daemon responses and +preserves the three fields, including explicit boolean values, in offline JSON. diff --git a/docs/completeness.md b/docs/completeness.md index ad51f2a..acfe877 100644 --- a/docs/completeness.md +++ b/docs/completeness.md @@ -11,7 +11,7 @@ partial or same-step stale response. ## Supported representation -Profile `sqlite-v1-format0-agy-1.3.4-5b93565c5f04-planner50-permission2-prompt23-tools143-permission1-tools144-skill1-history2-ide225-options145-compaction146-skill146-finish147-replay147-failedread146-subagents148-media149-tools149-sendmessage149-files1411-media1410-parentmsgs1410-paths1410-denied1412-cwd1412-unrun1412-config150-skill150-frontmatter150-bgcancel150-readlines150-config152-tools153-tools154-tools155-images155-tools156-retained156-legacychoices156-runtime157-planexecute157-tools158-workflow159` supports SQLite +Profile `sqlite-v1-format0-agy-1.3.4-5b93565c5f04-planner50-permission2-prompt23-tools143-permission1-tools144-skill1-history2-ide225-options145-compaction146-skill146-finish147-replay147-failedread146-subagents148-media149-tools149-sendmessage149-files1411-media1410-parentmsgs1410-paths1410-denied1412-cwd1412-unrun1412-config150-skill150-frontmatter150-bgcancel150-readlines150-config152-tools153-tools154-tools155-images155-tools156-retained156-legacychoices156-runtime157-planexecute157-tools158-workflow159-schedule1510-policy1510` supports SQLite `user_version=1`, the seven known tables and their exact columns, contiguous zero-based indices, and `step_format=0`. All persisted protobuf fields and enum values must be known, except the declared source-only fields below. @@ -1748,3 +1748,11 @@ the MIME type, and the stored byte count matches the inline data. The projected step preserves the filename, full byte sequence and MIME type. Markdown names the media and reports its size. See the [1.5.10 audit](compatibility/agy-1.5.10.md) for the native evidence and its limits. + +The `policy1510` profile adds `PolicyGuardianConfig.deployment` (string +field 15), `enable_irreversible_vetting` (optional boolean field 18), and +`enable_reversibility_adjudication` (optional boolean field 19). The fields +occur together in a stored command assessor configuration in executor +metadata. The full and ordinary responses preserve all three, including +explicit boolean presence. SQLite verification and offline JSON retain +them under the same equality rules as the surrounding metadata. diff --git a/internal/cache/sidecar.go b/internal/cache/sidecar.go index b9647e4..f5e08a1 100644 --- a/internal/cache/sidecar.go +++ b/internal/cache/sidecar.go @@ -23,6 +23,34 @@ import ( "github.com/mjacobs/agy-reader/internal/daemon" ) +// Capture publishes a provisional transcript only in an empty destination. +func Capture(path string, t *daemon.Trajectory, id, reason string, stamp time.Time) (bool, error) { + if t == nil || t.FromCache || id == "" || id != t.CascadeID || reason == "" { + return false, errors.New("cannot capture unidentified presentation") + } + var top map[string]json.RawMessage + if err := json.Unmarshal(t.RawJSON, &top); err != nil { + return false, err + } + if top == nil { + return false, errors.New("missing presentation") + } + reader, err := json.Marshal(map[string]any{"completeness": map[string]string{"state": "unverified", "reason": reason}}) + if err != nil { + return false, err + } + top["agyReader"] = reader + data, err := json.MarshalIndent(top, "", " ") + if err != nil { + return false, err + } + err = publishAtomic(path, append(data, '\n'), stamp, os.Chtimes, os.Link) + if errors.Is(err, fs.ErrExist) { + return false, nil + } + return err == nil, err +} + // Write atomically writes a trajectory to sidecarPath. Parent dir must // already exist (we don't create directories outside the supplied path). // @@ -219,6 +247,11 @@ func setParentCascadeID(sidecarPath, parentCascadeID string) (changed bool, err if err != nil { return false, err } + // A provisional file may be promoted concurrently. Never republish its + // bytes from this read-modify-write path. + if a := completeness.ReadAttestation(data); a != nil && a.State == "unverified" { + return false, nil + } var top map[string]json.RawMessage if err := json.Unmarshal(data, &top); err != nil { return false, fmt.Errorf("unmarshal sidecar for parent stamp: %w", err) @@ -333,10 +366,15 @@ func writeAtomicWithStamp(path string, data []byte, mtime time.Time, stamp func( return false, false, fmt.Errorf("read existing sidecar: %w", readErr) } + err = publishAtomic(path, data, mtime, stamp, os.Rename) + return err == nil, err == nil, err +} + +func publishAtomic(path string, data []byte, mtime time.Time, stamp func(string, time.Time, time.Time) error, publish func(string, string) error) error { dir := filepath.Dir(path) tmp, err := os.CreateTemp(dir, ".trajectory-*.json.tmp") if err != nil { - return false, false, fmt.Errorf("create temp sidecar: %w", err) + return fmt.Errorf("create temp sidecar: %w", err) } tmpPath := tmp.Name() cleanup := func() { @@ -345,27 +383,28 @@ func writeAtomicWithStamp(path string, data []byte, mtime time.Time, stamp func( } if err := tmp.Chmod(0o600); err != nil { cleanup() - return false, false, fmt.Errorf("chmod temp sidecar: %w", err) + return fmt.Errorf("chmod temp sidecar: %w", err) } if _, err := tmp.Write(data); err != nil { cleanup() - return false, false, fmt.Errorf("write temp sidecar: %w", err) + return fmt.Errorf("write temp sidecar: %w", err) } if err := stamp(tmpPath, mtime, mtime); err != nil { cleanup() - return false, false, fmt.Errorf("stamp temp sidecar mtime: %w", err) + return fmt.Errorf("stamp temp sidecar mtime: %w", err) } if err := tmp.Sync(); err != nil { cleanup() - return false, false, fmt.Errorf("sync temp sidecar: %w", err) + return fmt.Errorf("sync temp sidecar: %w", err) } if err := tmp.Close(); err != nil { _ = os.Remove(tmpPath) - return false, false, fmt.Errorf("close temp sidecar: %w", err) + return fmt.Errorf("close temp sidecar: %w", err) } - if err := os.Rename(tmpPath, path); err != nil { + if err := publish(tmpPath, path); err != nil { _ = os.Remove(tmpPath) - return false, false, fmt.Errorf("rename sidecar: %w", err) + return fmt.Errorf("publish sidecar: %w", err) } - return true, true, nil + _ = os.Remove(tmpPath) + return nil } diff --git a/internal/cache/sidecar_test.go b/internal/cache/sidecar_test.go index 2c4c478..371ffe5 100644 --- a/internal/cache/sidecar_test.go +++ b/internal/cache/sidecar_test.go @@ -476,3 +476,29 @@ func TestWriteVerifiedDoesNotRewindMtimeForPermissionOnlyRepair(t *testing.T) { t.Fatalf("mtime = %v, want the existing newer mtime %v", info.ModTime(), newer) } } + +func TestConcurrentCaptureKeepsVerifiedPublication(t *testing.T) { + p := filepath.Join(t.TempDir(), "a.trajectory.json") + traj := &daemon.Trajectory{CascadeID: "a", RawJSON: []byte(`{"cascadeId":"a","steps":[]}`)} + stamp := time.Now() + done := make(chan error) + go func() { _, err := cache.Capture(p, traj, "a", "unsupported projection", stamp); done <- err }() + verified := []byte("verified payload") + tmp := p + ".verified" + if err := os.WriteFile(tmp, verified, 0600); err != nil { + t.Fatal(err) + } + if err := os.Rename(tmp, p); err != nil { + t.Fatal(err) + } + if err := <-done; err != nil { + t.Fatal(err) + } + if created, err := cache.Capture(p, traj, "a", "unsupported projection", stamp); err != nil || created { + t.Fatal(created, err) + } + got, err := os.ReadFile(p) + if err != nil || !bytes.Equal(got, verified) { + t.Fatal(string(got), err) + } +} diff --git a/internal/completeness/apply_overlays.py b/internal/completeness/apply_overlays.py index 1d2488e..869d09c 100644 --- a/internal/completeness/apply_overlays.py +++ b/internal/completeness/apply_overlays.py @@ -262,6 +262,19 @@ class OverlayStep: compressed_bytes=138810, gzip_os=3, ), + OverlayStep( + name="policy1510", + module_name="generate_policy1510", + snapshot_path="artifacts/proto-snapshots/agy-1.5.10-20d6f74672ce/closure.protoset", + snapshot_kind="protoset", + snapshot_sha256="5e2adf25951e87fe151b3fd097ab2c772990ecfb167f3b2025cab40119aae1b1", + binary_sha256=None, + base_raw_sha256="f142a51963d6af1df06072e4c35d92be4a75dce1c545eeb421a7d4837b9483a2", + raw_sha256="07b862d0d5e8a4ed511f2d3e4d703148f87e2b29c72d32d055259b0a7316fd11", + compressed_sha256="d4d50760ceaff4e64e79c14d1b3f37ea8f5a8559574bbdddfd556c212e15fa23", + compressed_bytes=138900, + gzip_os=3, + ), ) @@ -478,7 +491,7 @@ def strip_step(step: OverlayStep, bundle): require(actual_in == step.raw_sha256, f"{step.name}: strip input hash mismatch") stripped = descriptor_pb2.FileDescriptorSet() stripped.CopyFrom(bundle) - if step.name in ("tools158", "workflow159", "schedule1510"): + if step.name in ("tools158", "workflow159", "schedule1510", "policy1510"): module = importlib.import_module(step.module_name) stripped = module.strip(stripped) elif step.name in _STRIPPERS: diff --git a/internal/completeness/attestation.go b/internal/completeness/attestation.go index 4af45ec..cef5ec7 100644 --- a/internal/completeness/attestation.go +++ b/internal/completeness/attestation.go @@ -17,6 +17,7 @@ import ( // SourceOnlyReason and SourceOnly records that content; both must match the // snapshot on every recheck. type Attestation struct { + State string `json:"state,omitempty"` Profile string `json:"profile"` SourceDigest string `json:"sourceDigest"` PayloadDigest string `json:"payloadDigest"` @@ -96,7 +97,7 @@ func (s *Snapshot) CheckAttestation(presentation []byte, a *Attestation) Result if s == nil { return Result{Unknown, "source snapshot unavailable"} } - if a == nil || a.Profile != s.Profile || a.Representation != representation { + if a == nil || a.State != "" || a.Profile != s.Profile || a.Representation != representation { return Result{Unknown, "missing or unsupported completeness attestation"} } digest, err := PayloadDigest(presentation) diff --git a/internal/completeness/generate_policy1510.py b/internal/completeness/generate_policy1510.py new file mode 100644 index 0000000..2959bd1 --- /dev/null +++ b/internal/completeness/generate_policy1510.py @@ -0,0 +1,58 @@ +"""Retain three observed PolicyGuardianConfig fields from the agy 1.5.10 snapshot.""" +import hashlib + +from google.protobuf import descriptor_pb2 +import generate_options145 as helpers +from generate_schedule1510 import SNAPSHOT, SNAPSHOT_HASH + +FIELDS = ( + ("deployment", 15, descriptor_pb2.FieldDescriptorProto.TYPE_STRING, False), + ("enable_irreversible_vetting", 18, descriptor_pb2.FieldDescriptorProto.TYPE_BOOL, True), + ("enable_reversibility_adjudication", 19, descriptor_pb2.FieldDescriptorProto.TYPE_BOOL, True), +) + + +def policy(bundle): + cortex = helpers.one(bundle.file, helpers.CORTEX_FILE) + return helpers.one(cortex.message_type, "PolicyGuardianConfig") + + +def augment(base, snapshot=SNAPSHOT): + raw = snapshot.read_bytes() + helpers.require(hashlib.sha256(raw).hexdigest() == SNAPSHOT_HASH, "snapshot hash mismatch") + source = descriptor_pb2.FileDescriptorSet.FromString(raw) + source.DiscardUnknownFields() + result = descriptor_pb2.FileDescriptorSet() + result.CopyFrom(base) + target_file, source_file = (helpers.one(bundle.file, helpers.CORTEX_FILE) for bundle in (result, source)) + helpers.require(target_file.package == source_file.package and + target_file.syntax == source_file.syntax == "proto3", + "source and target file semantics differ") + target, original = (policy(bundle) for bundle in (result, source)) + + for name, number, kind, has_presence in FIELDS: + field = helpers.one(original.field, name) + helpers.require((field.number, field.type, field.proto3_optional, field.HasField("oneof_index")) == + (number, kind, has_presence, has_presence), f"unexpected {name} declaration") + helpers.require(not any(f.name == name or f.number == number for f in target.field), + f"{name} already declared") + added = target.field.add() + added.CopyFrom(field) + if has_presence: + oneof = original.oneof_decl[field.oneof_index] + helpers.require(oneof.name == "_" + name, f"unexpected {name} presence declaration") + added.oneof_index = len(target.oneof_decl) + target.oneof_decl.add().CopyFrom(oneof) + + helpers.load_pool(result) + return result + + +def strip(bundle): + message = policy(bundle) + for name, _, _, has_presence in reversed(FIELDS): + message.field.remove(helpers.one(message.field, name)) + if has_presence: + helpers.require(message.oneof_decl[-1].name == "_" + name, f"unexpected {name} presence declaration") + del message.oneof_decl[-1] + return bundle diff --git a/internal/completeness/legacy.go b/internal/completeness/legacy.go index 2fd15fe..89ddafc 100644 --- a/internal/completeness/legacy.go +++ b/internal/completeness/legacy.go @@ -92,12 +92,20 @@ func readLegacySnapshot(ctx context.Context, path string) (*Snapshot, error) { if err != nil { return nil, refusal("legacy source authentication failed") } - m, err := decode("gemini_coder.Trajectory", plain) + m, err := newMessage("gemini_coder.Trajectory") if err != nil { return nil, refuse(err) } + if err := unmarshalRecord(plain, m); err != nil { + return nil, refusal("invalid persisted protobuf") + } s, err := finishSnapshotProfile(m, LegacyProfile) if err != nil { + msg := m.ProtoReflect() + id := msg.Get(msg.Descriptor().Fields().ByName("cascade_id")).String() + if id != "" && isShapeError(err) { + return nil, &UnknownShape{refuse(err), id, msg.Get(msg.Descriptor().Fields().ByName("steps")).List().Len()} + } return nil, refuse(err) } // Bind evidence to the exact authenticated file, including its nonce and tag. diff --git a/internal/completeness/policy1510_test.go b/internal/completeness/policy1510_test.go new file mode 100644 index 0000000..f2c99c8 --- /dev/null +++ b/internal/completeness/policy1510_test.go @@ -0,0 +1,46 @@ +package completeness + +import ( + "path/filepath" + "testing" + + "github.com/mjacobs/agy-reader/internal/testfixture" +) + +func TestPolicy1510RetainedAndCompared(t *testing.T) { + raw := `{"cascadeId":"fixture","metadata":{},"steps":[],"executorMetadatas":[` + + `{"cascadeConfig":{"executorConfig":{"commandAssessorConfig":{"policyGuardianConfig":` + + `{"deployment":"fixture","enableIrreversibleVetting":false,"enableReversibilityAdjudication":true}` + + `}}}}]}` + path := filepath.Join(t.TempDir(), "source.db") + testfixture.Source(t, path, []byte(raw)) + snapshot, err := ReadSnapshot(t.Context(), path) + if err != nil { + t.Fatal(err) + } + attestation, result := snapshot.VerifyResponses([]byte(raw), []byte(raw)) + if result.State != Verified || len(attestation.SourceOnly) != 0 { + t.Fatalf("policy fields did not compare normally: %+v", result) + } + for _, field := range []string{`"deployment":"fixture",`, `"enableIrreversibleVetting":false,`, `,"enableReversibilityAdjudication":true`} { + t.Run(field, func(t *testing.T) { + changed := replace1216(t, raw, field, "") + require1216(t, snapshot, changed, raw, Incomplete) + require1216(t, snapshot, raw, changed, Incomplete) + }) + } + offline, err := snapshot.OfflineJSON() + if err != nil { + t.Fatal(err) + } + require1216(t, snapshot, raw, string(offline), Verified) +} + +func TestPolicy1510RejectsUnqualifiedWireFields(t *testing.T) { + // Only the observed fields are added. In particular, field 20 stays unknown. + for _, data := range [][]byte{options145Varint(20, 1), options145Varint(15, 1), bytesField(18, 'x')} { + if _, err := decode("exa.cortex_pb.PolicyGuardianConfig", data); err == nil { + t.Fatal("unqualified policy wire field accepted") + } + } +} diff --git a/internal/completeness/refusal.go b/internal/completeness/refusal.go index 7381b33..8c938f9 100644 --- a/internal/completeness/refusal.go +++ b/internal/completeness/refusal.go @@ -77,3 +77,24 @@ func (s *Snapshot) Refusal() error { _, err := project(s.trajectory) return refuse(err) } + +// UnknownShape identifies a readable trajectory whose persisted fields exceed +// the profile. Its identity comes from the source that produced the refusal. +type UnknownShape struct { + Err error + CascadeID string + StepCount int +} + +func (e *UnknownShape) Error() string { return e.Err.Error() } +func (e *UnknownShape) Unwrap() error { return e.Err } + +// shapeError marks format extensions separately from malformed source data. +type shapeError string + +func (e shapeError) Error() string { return string(e) } + +func isShapeError(err error) bool { + var shape shapeError + return errors.Is(err, errUnknownFields) || errors.As(err, &shape) +} diff --git a/internal/completeness/schema.go b/internal/completeness/schema.go index 05fbdd1..525f07f 100644 --- a/internal/completeness/schema.go +++ b/internal/completeness/schema.go @@ -138,7 +138,8 @@ import ( // their scripts, results, render information or errors. // schedule1510 retains the timer countdown and qualifies captured running and // completed timer presentations and inline PDF reads. -const Profile = "sqlite-v1-format0-agy-1.3.4-5b93565c5f04-planner50-permission2-prompt23-tools143-permission1-tools144-skill1-history2-ide225-options145-compaction146-skill146-finish147-replay147-failedread146-subagents148-media149-tools149-sendmessage149-files1411-media1410-parentmsgs1410-paths1410-denied1412-cwd1412-unrun1412-config150-skill150-frontmatter150-bgcancel150-readlines150-config152-tools153-tools154-tools155-images155-tools156-retained156-legacychoices156-runtime157-planexecute157-tools158-workflow159-schedule1510" +// policy1510 retains the three PolicyGuardianConfig fields observed in executor metadata. +const Profile = "sqlite-v1-format0-agy-1.3.4-5b93565c5f04-planner50-permission2-prompt23-tools143-permission1-tools144-skill1-history2-ide225-options145-compaction146-skill146-finish147-replay147-failedread146-subagents148-media149-tools149-sendmessage149-files1411-media1410-parentmsgs1410-paths1410-denied1412-cwd1412-unrun1412-config150-skill150-frontmatter150-bgcancel150-readlines150-config152-tools153-tools154-tools155-images155-tools156-retained156-legacychoices156-runtime157-planexecute157-tools158-workflow159-schedule1510-policy1510" const maxBytes = 64 << 20 const maxDepth = 100 @@ -262,7 +263,7 @@ func normalizeIn(m protoreflect.Message, depth int, scope sourceOnlyScope) error name := typ[strings.LastIndex(typ, "/")+1:] mt, err := schemaTypes.FindMessageByName(protoreflect.FullName(name)) if err != nil { - return errors.New("unsupported Any payload type") + return shapeError("unsupported Any payload type") } nested := mt.New().Interface() if err = unmarshalRecord(data, nested); err != nil { @@ -290,7 +291,7 @@ func normalizeIn(m protoreflect.Message, depth int, scope sourceOnlyScope) error return normalizeIn(v.Message(), depth+1, next) case protoreflect.EnumKind: if f.Enum().Values().ByNumber(v.Enum()) == nil { - return fmt.Errorf("unsupported enum in %s", f.FullName()) + return shapeError("unsupported enum in " + string(f.FullName())) } } return nil diff --git a/internal/completeness/schema.pb.gz b/internal/completeness/schema.pb.gz index 34d72e6..63af98f 100644 Binary files a/internal/completeness/schema.pb.gz and b/internal/completeness/schema.pb.gz differ diff --git a/internal/completeness/sqlite.go b/internal/completeness/sqlite.go index a85df61..3bb0a8a 100644 --- a/internal/completeness/sqlite.go +++ b/internal/completeness/sqlite.go @@ -61,7 +61,7 @@ func ReadSnapshot(ctx context.Context, path string) (*Snapshot, error) { return s, err } -func readSnapshot(ctx context.Context, path string) (*Snapshot, error) { +func readSnapshot(ctx context.Context, path string) (_ *Snapshot, resultErr error) { path, err := filepath.Abs(path) if err != nil { return nil, err @@ -126,6 +126,21 @@ func readSnapshot(ctx context.Context, path string) (*Snapshot, error) { if count != 1 { return nil, refusal("source must contain exactly one trajectory") } + defer func() { + if resultErr == nil || !isShapeError(resultErr) { + return + } + cascade := out.Get(fields.ByName("cascade_id")).String() + if cascade == "" { + return + } + // A decode refusal may interrupt readSteps before the list is complete. + steps := out.Get(fields.ByName("steps")).List().Len() + if err := tx.QueryRowContext(ctx, "SELECT count(*) FROM steps").Scan(&steps); err != nil && !errors.Is(err, sql.ErrTxDone) { + return + } + resultErr = &UnknownShape{resultErr, cascade, steps} + }() budget := maxBytes if err = readSteps(ctx, tx, out, &budget); err != nil { return nil, err @@ -268,10 +283,10 @@ func readSteps(ctx context.Context, tx *sql.Tx, out protoreflect.Message, budget } index++ if format != 0 { - return refusal("unsupported step format") + return refuse(shapeError("unsupported step format")) } if sub { - return refusal("unsupported decomposed step representation") + return refuse(shapeError("unsupported decomposed step representation")) } *budget -= len(payload) + len(metadata) + len(details) + len(permissions) + len(task) + len(render) if *budget < 0 { diff --git a/internal/completeness/tools158_test.go b/internal/completeness/tools158_test.go index 5f0184b..d94fda3 100644 --- a/internal/completeness/tools158_test.go +++ b/internal/completeness/tools158_test.go @@ -158,7 +158,8 @@ func TestCurrentProfileInvalidatesPreviousAttestations(t *testing.T) { for _, previous := range []string{ strings.Split(Profile, "-tools158")[0], strings.Split(Profile, "-workflow159")[0], - strings.TrimSuffix(Profile, "-schedule1510"), + strings.Split(Profile, "-schedule1510")[0], + strings.TrimSuffix(Profile, "-policy1510"), } { a.Profile = previous if a.Profile == Profile { diff --git a/internal/contract/trajectory-sidecar.schema.json b/internal/contract/trajectory-sidecar.schema.json index 1afce75..1cfc735 100644 --- a/internal/contract/trajectory-sidecar.schema.json +++ b/internal/contract/trajectory-sidecar.schema.json @@ -80,6 +80,27 @@ "description": "Bare UUID of the immediate parent cascade." }, "completeness": { + "description": "Verified projection evidence or a provisional capture marked unverified.", + "if": { + "required": [ + "state" + ] + }, + "then": { + "required": [ + "reason" + ], + "properties": { + "state": { + "const": "unverified" + }, + "reason": { + "type": "string", + "minLength": 1 + } + }, + "additionalProperties": false + }, "type": "object", "additionalProperties": true } diff --git a/internal/discovery/cli_address.go b/internal/discovery/cli_address.go new file mode 100644 index 0000000..759eef7 --- /dev/null +++ b/internal/discovery/cli_address.go @@ -0,0 +1,38 @@ +package discovery + +import ( + "net" + "strconv" + "strings" +) + +// localAddressKey accepts only literal loopback endpoints (localhost retains +// its historical IPv4 meaning). Credentials never cross address families merely +// because two listeners have the same port number. +func localAddressKey(address string) string { + host, port, err := net.SplitHostPort(address) + if err != nil { + return "" + } + switch host { + case "localhost", "127.0.0.1": + host = "127.0.0.1" + case "::1": + default: + return "" + } + n, err := strconv.ParseUint(port, 10, 16) + if err != nil || n == 0 { + return "" + } + return net.JoinHostPort(host, strconv.FormatUint(n, 10)) +} + +func sameLocalEndpoint(a, b string) bool { + const prefix = "http://" + if !strings.HasPrefix(a, prefix) || !strings.HasPrefix(b, prefix) { + return false + } + address := localAddressKey(strings.TrimSuffix(strings.TrimPrefix(a, prefix), "/")) + return address != "" && address == localAddressKey(strings.TrimPrefix(b, prefix)) +} diff --git a/internal/discovery/cli_darwin.go b/internal/discovery/cli_darwin.go new file mode 100644 index 0000000..24b6341 --- /dev/null +++ b/internal/discovery/cli_darwin.go @@ -0,0 +1,227 @@ +package discovery + +import ( + "bytes" + "cmp" + "context" + "encoding/binary" + "os" + "os/exec" + "path/filepath" + "slices" + "strconv" + "strings" + "time" + + "golang.org/x/sys/unix" +) + +func discoverCLIEndpoint(root, wantedURL string) (string, string) { + if found := darwinCLIConnections(root, wantedURL); len(found) != 0 { + return found[0].BaseURL, found[0].Token + } + return "", "" +} + +func discoverCLIConnections(root string) []Connection { return darwinCLIConnections(root, "") } + +func darwinCLIConnections(root, wantedURL string) []Connection { + root, err := filepath.EvalSymlinks(root) + if err != nil { + return nil + } + root, err = filepath.Abs(root) + if err != nil { + return nil + } + uid := os.Getuid() + processes, err := unix.SysctlKinfoProcSlice("kern.proc.uid", uid) + if err != nil { + return nil + } + children := make(map[int32][]unix.KinfoProc) + var cli []unix.KinfoProc + for _, p := range processes { + if p.Eproc.Pcred.P_ruid != uint32(uid) || p.Proc.P_pid <= 0 { + continue + } + children[p.Eproc.Ppid] = append(children[p.Eproc.Ppid], p) + if unix.ByteSliceToString(p.Proc.P_comm[:]) == "agy" { + cli = append(cli, p) + } + } + slices.SortFunc(cli, func(a, b unix.KinfoProc) int { + return cmp.Compare(b.Proc.P_starttime.Nano(), a.Proc.P_starttime.Nano()) + }) + + var authenticated, pending []Connection + for _, p := range cli { + listeners := darwinCLIListeners(root, p.Proc.P_pid) + for _, address := range listeners { + base := "http://" + address + if wantedURL != "" && !sameLocalEndpoint(wantedURL, base) { + continue + } + token := darwinCLIToolToken(p.Proc.P_pid, children, address) + if !sameDarwinProcess(p) { + break + } + conn := Connection{BaseURL: base, Token: token} + if token != "" { + authenticated = append(authenticated, conn) + } else { + pending = append(pending, conn) + } + } + } + return append(authenticated, pending...) +} + +// Use PID selection because lsof may truncate the executable's name. Accept +// only a loopback TCP listener whose port is declared HTTP in a CLI log +// this process has open inside the requested store. +func darwinCLIListeners(root string, pid int32) []string { + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + data, err := exec.CommandContext(ctx, "/usr/sbin/lsof", "-nPa", "-p", strconv.Itoa(int(pid)), "-FfntT0").Output() + if err != nil { + return nil + } + return darwinCLIHTTPListeners(root, data) +} + +func darwinCLIHTTPListeners(root string, data []byte) []string { + var name, fileType string + var descriptor, listening bool + var listeners []string + ports := make(map[string]bool) + logs := make(map[string]bool) + finishFile := func() { + if !descriptor { + return + } + if listening && (fileType == "IPv4" || fileType == "IPv6") { + if address := localAddressKey(name); address != "" { + listeners = append(listeners, address) + } + } + if fileType == "REG" && !logs[name] && (name == filepath.Join(root, "cli.log") || + (filepath.Dir(name) == filepath.Join(root, "log") && strings.HasPrefix(filepath.Base(name), "cli-") && strings.HasSuffix(name, ".log"))) { + logs[name] = true + if log, err := os.ReadFile(name); err == nil { + if port, ok := parseDaemonPort(string(log)); ok { + ports[port] = true + } + } + } + } + for _, field := range strings.Split(string(data), "\x00") { + field = strings.TrimPrefix(field, "\n") + if field == "" { + continue + } + switch field[0] { + case 'f': + finishFile() + _, err := strconv.Atoi(field[1:]) + descriptor = err == nil + name, fileType, listening = "", "", false + case 't': + fileType = field[1:] + case 'n': + name = field[1:] + case 'T': + listening = listening || field == "TST=LISTEN" + } + } + finishFile() + return slices.DeleteFunc(listeners, func(address string) bool { + return !ports[address[strings.LastIndexByte(address, ':')+1:]] + }) +} + +// Same-user descendants can carry the credentials agy delegates to tools. +// The watcher never probes unrelated environments or crosses daemon endpoints. +func darwinCLIToolToken(pid int32, children map[int32][]unix.KinfoProc, address string) string { + queue := append([]unix.KinfoProc(nil), children[pid]...) + token := "" + for len(queue) > 0 { + p := queue[0] + queue = queue[1:] + if p.Proc.P_pid == pid || !sameDarwinProcess(p) { + continue + } + queue = append(queue, children[p.Proc.P_pid]...) + data, err := unix.SysctlRaw("kern.procargs2", int(p.Proc.P_pid)) + if err != nil || !sameDarwinProcess(p) { + continue + } + candidate := darwinCLIEnvironmentToken(data, address) + if candidate == "" { + continue + } + if token != "" && token != candidate { + return "" + } + token = candidate + } + return token +} + +func sameDarwinProcess(p unix.KinfoProc) bool { + current, err := unix.SysctlKinfoProc("kern.proc.pid", int(p.Proc.P_pid)) + return err == nil && current.Proc.P_starttime == p.Proc.P_starttime && + current.Eproc.Ppid == p.Eproc.Ppid && current.Eproc.Pcred.P_ruid == p.Eproc.Pcred.P_ruid && + current.Eproc.Ucred.Uid == p.Eproc.Ucred.Uid +} + +// KERN_PROCARGS2 starts with argc, the executable path and alignment padding, +// then argc NUL-terminated arguments followed by the initial environment. +func darwinCLIEnvironmentToken(data []byte, address string) string { + if len(data) < 4 { + return "" + } + argc := int(binary.NativeEndian.Uint32(data[:4])) + _, args, ok := bytes.Cut(data[4:], []byte{0}) + if !ok || argc <= 0 { + return "" + } + args = bytes.TrimLeft(args, "\x00") + for range argc { + _, args, ok = bytes.Cut(args, []byte{0}) + if !ok { + return "" + } + } + var endpoint, token, version string + for len(args) > 0 && args[0] != 0 { + var entry []byte + entry, args, ok = bytes.Cut(args, []byte{0}) + if !ok { + return "" + } + key, value, hasValue := bytes.Cut(entry, []byte{'='}) + if !hasValue { + continue + } + var field *string + switch string(key) { + case "ANTIGRAVITY_LS_ADDRESS": + field = &endpoint + case "ANTIGRAVITY_CSRF_TOKEN": + field = &token + case "ANTIGRAVITY_LS_VERSION": + field = &version + default: + continue + } + if *field != "" || len(value) == 0 { + return "" + } + *field = string(value) + } + if !strings.HasPrefix(version, "cli-") || localAddressKey(endpoint) != address { + return "" + } + return token +} diff --git a/internal/discovery/cli_darwin_test.go b/internal/discovery/cli_darwin_test.go new file mode 100644 index 0000000..7b483b2 --- /dev/null +++ b/internal/discovery/cli_darwin_test.go @@ -0,0 +1,57 @@ +package discovery + +import ( + "encoding/binary" + "os" + "path/filepath" + "slices" + "strings" + "testing" +) + +func TestDarwinCLIHTTPListeners(t *testing.T) { + root := t.TempDir() + logPath := filepath.Join(root, "cli.log") + if err := os.WriteFile(logPath, []byte("listening on random port at 45001 for HTTP\nlistening on random port at 45002 for HTTPS (gRPC)\n"), 0o600); err != nil { + t.Fatal(err) + } + fields := strings.Join([]string{ + "p101", "\nf2", "tREG", "n" + logPath, + "\nf3", "tIPv4", "n127.0.0.1:45002", "TST=LISTEN", + "\nf4", "tIPv4", "n127.0.0.1:45001", "TST=ESTABLISHED", + "\nf5", "tIPv4", "n0.0.0.0:45001", "TST=LISTEN", + "\nf6", "tIPv6", "n[::1]:45001", "TST=LISTEN", "\n", + }, "\x00") + if got := darwinCLIHTTPListeners(root, []byte(fields)); !slices.Equal(got, []string{"[::1]:45001"}) { + t.Fatalf("HTTP listeners = %v, want only the logged loopback listener", got) + } + if got := darwinCLIHTTPListeners(t.TempDir(), []byte(fields)); len(got) > 0 { + t.Fatalf("returned %d listeners for a different root", len(got)) + } +} + +func TestDarwinCLIEnvironmentToken(t *testing.T) { + const address = "127.0.0.1:45001" + const toolAddress = "ANTIGRAVITY_LS_ADDRESS=localhost:45001\x00" + const toolToken = "ANTIGRAVITY_CSRF_TOKEN=fixture-token\x00" + const toolVersion = "ANTIGRAVITY_LS_VERSION=cli-fixture\x00" + args := binary.NativeEndian.AppendUint32(nil, 3) + args = append(args, "/bin/sh\x00\x00\x00sh\x00"+toolToken+"\x00"...) + for _, tc := range []struct { + name, env, want string + }{ + {"CLI tool", toolAddress + toolVersion + toolToken, "fixture-token"}, + {"argv token only", toolAddress + toolVersion, ""}, + {"wrong port", "ANTIGRAVITY_LS_ADDRESS=localhost:45002\x00" + toolVersion + toolToken, ""}, + {"wrong address family", "ANTIGRAVITY_LS_ADDRESS=[::1]:45001\x00" + toolVersion + toolToken, ""}, + {"non-CLI tool", toolAddress + toolToken, ""}, + {"duplicate token", toolAddress + toolVersion + toolToken + toolToken, ""}, + {"truncated token", toolAddress + toolVersion + strings.TrimSuffix(toolToken, "\x00"), ""}, + } { + t.Run(tc.name, func(t *testing.T) { + if got := darwinCLIEnvironmentToken(append(slices.Clone(args), tc.env...), address); got != tc.want { + t.Fatal("unexpected credential discovery result") + } + }) + } +} diff --git a/internal/discovery/cli_linux.go b/internal/discovery/cli_linux.go index 7c6673d..11b5a9f 100644 --- a/internal/discovery/cli_linux.go +++ b/internal/discovery/cli_linux.go @@ -297,34 +297,3 @@ func listeningEndpoints(dir string, sockets map[string]bool) []localListener { } return listeners } - -// localAddressKey accepts only literal loopback endpoints (localhost retains -// its historical IPv4 meaning). Credentials never cross address families merely -// because two listeners have the same port number. -func localAddressKey(address string) string { - host, port, err := net.SplitHostPort(address) - if err != nil { - return "" - } - switch host { - case "localhost", "127.0.0.1": - host = "127.0.0.1" - case "::1": - default: - return "" - } - n, err := strconv.ParseUint(port, 10, 16) - if err != nil || n == 0 { - return "" - } - return net.JoinHostPort(host, strconv.FormatUint(n, 10)) -} - -func sameLocalEndpoint(a, b string) bool { - const prefix = "http://" - if !strings.HasPrefix(a, prefix) || !strings.HasPrefix(b, prefix) { - return false - } - address := localAddressKey(strings.TrimSuffix(strings.TrimPrefix(a, prefix), "/")) - return address != "" && address == localAddressKey(strings.TrimPrefix(b, prefix)) -} diff --git a/internal/discovery/cli_other.go b/internal/discovery/cli_other.go index 2d2bfd3..8d7edd1 100644 --- a/internal/discovery/cli_other.go +++ b/internal/discovery/cli_other.go @@ -1,4 +1,4 @@ -//go:build !linux +//go:build !linux && !darwin package discovery diff --git a/internal/doctor/doctor.go b/internal/doctor/doctor.go index 0aafa3b..59bdfa6 100644 --- a/internal/doctor/doctor.go +++ b/internal/doctor/doctor.go @@ -1,6 +1,7 @@ package doctor import ( + "bytes" "context" _ "embed" "errors" @@ -68,6 +69,8 @@ type Report struct { Stale int StaleIDs []string // up to 5 missing/stale cascade IDs SQLite, Verified, Incomplete, Unknown int + SavedUnverified int + UnreadableSidecars int Unsupported int UnsupportedSessions []string // up to 5 unsupported sessions (" ()") Unreadable int // Sources that could not be read now (lock, access) @@ -107,7 +110,7 @@ func WriteMultiReport(w io.Writer, reports []Report, explicit bool) int { if WriteReportBody(w, r) > 0 { unhealthy++ } - authRejected = authRejected || r.AuthRejected || r.Incomplete > 0 || r.Unknown > 0 + authRejected = authRejected || r.AuthRejected || r.Incomplete > 0 || r.Unknown > 0 || r.SavedUnverified > 0 || r.UnreadableSidecars > 0 } fail := unhealthy == len(reports) if explicit { @@ -215,6 +218,14 @@ func WriteReportBody(w io.Writer, r Report) int { } // sidecars + if r.UnreadableSidecars > 0 { + problems++ + fmt.Fprintf(w, " transcripts: %d unreadable -> check sidecar file access\n", r.UnreadableSidecars) + } + if r.SavedUnverified > 0 { + problems++ + fmt.Fprintf(w, " transcripts: %d saved unverified (verification needs a supported source and daemon)\n", r.SavedUnverified) + } switch { case r.CoverageErr != nil: // Could not enumerate sessions (e.g. an unreadable conversations dir). @@ -521,7 +532,35 @@ type sidecarCheck struct { func PopulateSidecarCoverage(r *Report, root string) { sessions, err := discovery.ListConversationSessions(root) r.CoverageErr = err - if err != nil || len(sessions) == 0 { + if err != nil { + return + } + dir := filepath.Join(root, "conversations") + entries, err := os.ReadDir(dir) + if err != nil && !errors.Is(err, os.ErrNotExist) { + r.CoverageErr = err + return + } + for _, entry := range entries { + if !entry.Type().IsRegular() || !strings.HasSuffix(entry.Name(), ".trajectory.json") { + continue + } + raw, err := os.ReadFile(filepath.Join(dir, entry.Name())) + if errors.Is(err, os.ErrNotExist) { + continue + } + if err != nil { + r.UnreadableSidecars++ + continue + } + if !bytes.Contains(raw, []byte(`"unverified"`)) { + continue + } + if a := completeness.ReadAttestation(raw); a != nil && a.State == "unverified" { + r.SavedUnverified++ + } + } + if len(sessions) == 0 { return } diff --git a/internal/doctor/doctor_test.go b/internal/doctor/doctor_test.go index c256a68..9494b92 100644 --- a/internal/doctor/doctor_test.go +++ b/internal/doctor/doctor_test.go @@ -732,3 +732,32 @@ func TestWriteReportBodySuppressesUnpinnedCSRFWhenDaemonUnreachable(t *testing.T t.Fatalf("unreachable CLI without PinnedCSRF must not report csrf line:\n%s", cliBuf.String()) } } + +func TestDoctorReportsUnverifiedAfterSourceRemoval(t *testing.T) { + root := t.TempDir() + writeFileT(t, filepath.Join(root, "conversations", "archived.trajectory.json"), []byte(`{"agyReader":{"completeness":{"state":"unverified","reason":"unsupported projection"}}}`)) + var r Report + PopulateSidecarCoverage(&r, root) + var out bytes.Buffer + if r.SavedUnverified != 1 || r.Fresh != 0 || WriteMultiReport(&out, []Report{r, {}}, false) != 1 || !strings.Contains(out.String(), "1 saved unverified") { + t.Fatal(r, out.String()) + } +} + +func TestUnreadableSidecarKeepsOtherCounts(t *testing.T) { + root := t.TempDir() + p := filepath.Join(root, "conversations", "hidden.trajectory.json") + writeFileT(t, p, []byte("{}")) + if err := os.Chmod(p, 0); err != nil { + t.Fatal(err) + } + if _, err := os.ReadFile(p); err == nil { + t.Skip("process can read mode 000") + } + writeFileT(t, filepath.Join(root, "conversations", "saved.trajectory.json"), []byte(`{"agyReader":{"completeness":{"state":"unverified","reason":"unsupported projection"}}}`)) + var r Report + PopulateSidecarCoverage(&r, root) + if r.CoverageErr != nil || r.UnreadableSidecars != 1 || r.SavedUnverified != 1 { + t.Fatal(r) + } +} diff --git a/internal/session/session.go b/internal/session/session.go index db08f1b..d3ba621 100644 --- a/internal/session/session.go +++ b/internal/session/session.go @@ -15,8 +15,11 @@ import ( ) type CompletenessError struct { - Result completeness.Result - BeforeRPC bool + Result completeness.Result + BeforeRPC bool + CaptureID string + CaptureSteps int + Presentation *daemon.Trajectory // SourceDigest is the digest of the source snapshot, set only when that // snapshot and the profile alone prove the refusal: an empty persisted // identity or a projection the profile does not support. Retrying the same @@ -51,7 +54,13 @@ func NewDaemonClient(root string, conn discovery.Connection) *daemon.Client { func Fetch(ctx context.Context, client *daemon.Client, s discovery.Session) (*daemon.Trajectory, error) { before, err := completeness.ReadSnapshot(ctx, s.PBPath) if err != nil { - return nil, &CompletenessError{Result: completeness.Result{State: completeness.Unknown, Reason: err.Error()}, BeforeRPC: true} + refusal := &CompletenessError{Result: completeness.Result{State: completeness.Unknown, Reason: err.Error()}, BeforeRPC: true} + var shape *completeness.UnknownShape + if errors.As(err, &shape) { + refusal.CaptureID = shape.CascadeID + refusal.CaptureSteps = shape.StepCount + } + return nil, refusal } if err = client.LoadTrajectory(ctx, s.CascadeID); err != nil { return nil, err @@ -73,6 +82,12 @@ func Fetch(ctx context.Context, client *daemon.Client, s discovery.Session) (*da refusal := &CompletenessError{Result: result} if sourceProven { refusal.SourceDigest = before.Digest + if presentation.CascadeID == id && len(presentation.Steps) == before.StepCount { + refusal.CaptureID = id + refusal.Presentation = presentation + } else { + return nil, Unverified(completeness.Incomplete, "daemon presentation identity or step count differs from source") + } } return nil, refusal } diff --git a/internal/watcher/refusals.go b/internal/watcher/refusals.go index 897190b..d32bb38 100644 --- a/internal/watcher/refusals.go +++ b/internal/watcher/refusals.go @@ -22,6 +22,7 @@ type RefusalMemo map[string]HeldRefusal var refusalNow = time.Now const mismatchRetryDelay = 30 * time.Second +const captureRetryDelay = 5 * time.Minute type HeldRefusal struct { RetryAt time.Time // transient RPC mismatch; never an indefinite source refusal @@ -32,6 +33,9 @@ type HeldRefusal struct { SidecarState SidecarState // SourceFiles is the source metadata at a transient mismatch, or when // its digest last matched SourceDigest at DigestMatchedAt. + // A bounded capture retry can use the discovery mtime on platforms + // without file fingerprints. Endpoint, profile and sidecar checks still apply. + MtimeOnlyRetry bool SourceFiles vcache.Files DigestMatchedAt time.Time } @@ -80,7 +84,7 @@ func HoldableRefusal(err error) (digest string, ok bool) { if !errors.As(err, &contentErr) || contentErr.Result.State != completeness.Unknown { return "", false } - if contentErr.BeforeRPC { + if contentErr.BeforeRPC || contentErr.CaptureID != "" && contentErr.SourceDigest == "" { return "", true } return contentErr.SourceDigest, contentErr.SourceDigest != "" @@ -135,7 +139,7 @@ func (m RefusalMemo) holds(s discovery.Session, endpoint string, readDigest func } if !held.RetryAt.IsZero() { files, ok := vcache.SourceFingerprint(s.PBPath) - if !ok || files != held.SourceFiles || !refusalNow().Before(held.RetryAt) { + if (!held.MtimeOnlyRetry && (!ok || files != held.SourceFiles)) || !refusalNow().Before(held.RetryAt) { delete(m, s.PBPath) return false } @@ -189,14 +193,18 @@ func (m RefusalMemo) Hold(s discovery.Session, endpoint, digest string) { // File, sidecar, endpoint and profile changes release it immediately. Even with // none of those changed, the same endpoint is retried after a short deadline. func (m RefusalMemo) HoldMismatch(s discovery.Session, endpoint string) { + m.holdUntilRetry(s, endpoint, mismatchRetryDelay) +} + +func (m RefusalMemo) holdUntilRetry(s discovery.Session, endpoint string, delay time.Duration) { if m == nil { return } files, ok := vcache.SourceFingerprint(s.PBPath) - if !ok { + if !ok && delay != captureRetryDelay { return } - m[s.PBPath] = HeldRefusal{Endpoint: endpoint, Profile: completeness.ProfileForPath(s.PBPath), SourceModTime: s.ModTime, SidecarState: StatSidecar(s.SidecarPath), SourceFiles: files, RetryAt: refusalNow().Add(mismatchRetryDelay)} + m[s.PBPath] = HeldRefusal{MtimeOnlyRetry: !ok, Endpoint: endpoint, Profile: completeness.ProfileForPath(s.PBPath), SourceModTime: s.ModTime, SidecarState: StatSidecar(s.SidecarPath), SourceFiles: files, RetryAt: refusalNow().Add(delay)} } func mismatchedResponse(err error) bool { diff --git a/internal/watcher/watch_refusals_test.go b/internal/watcher/watch_refusals_test.go index 7de6ff3..fe0d4bc 100644 --- a/internal/watcher/watch_refusals_test.go +++ b/internal/watcher/watch_refusals_test.go @@ -3,11 +3,14 @@ package watcher import ( "bytes" "os" + "path/filepath" "runtime" "strings" "testing" "time" + "github.com/mjacobs/agy-reader/internal/cache" + "github.com/mjacobs/agy-reader/internal/completeness" "github.com/mjacobs/agy-reader/internal/daemon" "github.com/mjacobs/agy-reader/internal/session" "github.com/mjacobs/agy-reader/internal/testfixture" @@ -47,8 +50,10 @@ func TestWatchHoldsUnchangedUnknownSourceWithoutDaemonCalls(t *testing.T) { if w.ConsecutiveFailures != 0 { t.Fatalf("reachable daemon counted as failing: %d", w.ConsecutiveFailures) } - if _, err := os.Stat(s.SidecarPath); !os.IsNotExist(err) { - t.Fatalf("refused source wrote a sidecar: %v", err) + raw, err := os.ReadFile(s.SidecarPath) + a := completeness.ReadAttestation(raw) + if err != nil || a == nil || a.State != "unverified" || a.Reason != "unsupported generic arguments" || a.Representation != "" { + t.Fatalf("missing provisional capture: %v %v", a, err) } // Content change, which the daemon also serves: retried on the very next @@ -355,3 +360,79 @@ func TestWatchTreatsForkedConversationAsOneDBSource(t *testing.T) { t.Fatalf("held source is not the .db %s: %v", s.PBPath, w.Held) } } + +// Exercise the refusal before the first RPC, then restart with a source whose +// fields the profile recognizes. Both ticks use the normal watcher entry point. +func TestUnknownPersistedFieldCaptureAndPromotion(t *testing.T) { + root, s, db := completeFixture(t) + var original []byte + if err := db.QueryRow("SELECT step_payload FROM steps WHERE idx=0").Scan(&original); err != nil { + t.Fatal(err) + } + testfixture.Exec(t, db, "UPDATE steps SET step_payload=? WHERE idx=0", append(bytes.Clone(original), 0xf8, 0x7f, 0x01)) + d := newPayloadDaemon(t, completePayload("current")) + var logs bytes.Buffer + heldWatcher(t, root, d.URL, &logs).Tick() + data, err := os.ReadFile(s.SidecarPath) + if err != nil { + t.Fatalf("%v %s", err, &logs) + } + a := completeness.ReadAttestation(data) + if a == nil || a.State != "unverified" || a.Reason != "unknown persisted protobuf fields" { + t.Fatal(a) + } + testfixture.Exec(t, db, "UPDATE steps SET step_payload=? WHERE idx=0", original) + heldWatcher(t, root, d.URL, &logs).Tick() + if got := session.Completeness(s); got.State != completeness.Verified { + t.Fatal(got, logs.String()) + } +} + +func TestRestartPromotesFirstCaptureWithUnchangedSource(t *testing.T) { + root, s, _ := completeFixture(t) + raw := completePayload("current") + _, err := cache.Capture(s.SidecarPath, &daemon.Trajectory{CascadeID: completeID, RawJSON: raw}, completeID, "previous profile refused this shape", s.ModTime) + if err != nil { + t.Fatal(err) + } + d := newPayloadDaemon(t, raw) + heldWatcher(t, root, d.URL, new(bytes.Buffer)).Tick() + if got := session.Completeness(s); got.State != completeness.Verified { + t.Fatal(got) + } +} + +func TestFailedFirstCaptureBacksOffAndRecovers(t *testing.T) { + root, s, _ := unsupportedProjectionFixture(t) + dir := filepath.Dir(s.SidecarPath) + if err := os.Chmod(dir, 0500); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.Chmod(dir, 0700) }) + if probe, err := os.CreateTemp(dir, "probe"); err == nil { + probe.Close() + t.Skip("process can write read-only directory") + } + d := newPayloadDaemon(t, unsupportedProjectionPayload("first")) + var logs bytes.Buffer + now := time.Now() + old := refusalNow + refusalNow = func() time.Time { return now } + t.Cleanup(func() { refusalNow = old }) + w := heldWatcher(t, root, d.URL, &logs) + w.Tick() + calls := d.calls.Load() + now = now.Add(time.Minute) + w.Tick() + if d.calls.Load() != calls || !strings.Contains(logs.String(), "retrying capture") { + t.Fatal(logs.String()) + } + if err := os.Chmod(dir, 0700); err != nil { + t.Fatal(err) + } + now = now.Add(captureRetryDelay) + w.Tick() + if _, err := os.Stat(s.SidecarPath); err != nil { + t.Fatal(err, logs.String()) + } +} diff --git a/internal/watcher/watcher.go b/internal/watcher/watcher.go index f7374dd..64db1c5 100644 --- a/internal/watcher/watcher.go +++ b/internal/watcher/watcher.go @@ -16,6 +16,7 @@ import ( "time" "github.com/mjacobs/agy-reader/internal/cache" + "github.com/mjacobs/agy-reader/internal/completeness" "github.com/mjacobs/agy-reader/internal/daemon" "github.com/mjacobs/agy-reader/internal/discovery" "github.com/mjacobs/agy-reader/internal/session" @@ -365,6 +366,12 @@ func WatchTickCachedPolicy(ctx context.Context, client *daemon.Client, root stri } endpoint := client.BaseURL traj, err := session.FetchRecover(ctx, client, root, s, pinned) + captureFailed := false + var refusal *session.CompletenessError + if errors.As(err, &refusal) && refusal.CaptureID != "" && !cache.Exists(s.SidecarPath) { + err = captureRefusal(ctx, client, s, refusal, logger) + captureFailed = err != refusal + } proxied := daemon.ClientProxied(client) if FetchContactedDaemon(err, proxied) { tickContactedDaemon = true @@ -409,6 +416,9 @@ func WatchTickCachedPolicy(ctx context.Context, client *daemon.Client, root stri } else if mismatchedResponse(err) && held != nil && ctx.Err() == nil { held.HoldMismatch(s, endpoint) logger.Printf("watch: skip %s (%v); retrying in %s unless its source or daemon endpoint changes", s.CascadeID, err, mismatchRetryDelay) + } else if captureFailed && held != nil && ctx.Err() == nil { + held.holdUntilRetry(s, endpoint, captureRetryDelay) + logger.Printf("watch: skip %s (%v); retrying capture in %s unless its inputs change", s.CascadeID, err, captureRetryDelay) } else { logger.Printf("watch: skip %s (%v)", s.CascadeID, err) } @@ -442,6 +452,40 @@ func WatchTickCachedPolicy(ctx context.Context, client *daemon.Client, root stri return } +// captureRefusal returns the original refusal after publication so the watcher +// still records its reason and releases the hold on a profile or source change. +func captureRefusal(ctx context.Context, client *daemon.Client, s discovery.Session, refusal *session.CompletenessError, logger *log.Logger) error { + t := refusal.Presentation + if t == nil { + if err := client.LoadTrajectory(ctx, s.CascadeID); err != nil { + return err + } + var err error + t, err = client.GetCascadeTrajectory(ctx, refusal.CaptureID) + if err != nil { + return err + } + } + after, err := completeness.ReadSnapshot(ctx, s.PBPath) + var shape *completeness.UnknownShape + if refusal.BeforeRPC && (t.CascadeID != refusal.CaptureID || len(t.Steps) != refusal.CaptureSteps) { + return session.Unverified(completeness.Incomplete, "daemon presentation identity or step count differs from source") + } + sameShape := errors.As(err, &shape) && shape.CascadeID == refusal.CaptureID && shape.StepCount == refusal.CaptureSteps && shape.Error() == refusal.Result.Reason && refusal.BeforeRPC + if !sameShape && (err != nil || after.Digest != refusal.SourceDigest) { + return session.Unverified(completeness.Incomplete, "persisted source changed during fetch") + } + saved, err := cache.Capture(s.SidecarPath, t, refusal.CaptureID, refusal.Result.Reason, s.ModTime) + if err != nil { + return err + } + if saved { + logger.Printf("watch: saved unverified transcript %s (%s)", s.CascadeID, refusal.Result.Reason) + } + refusal.BeforeRPC = false + return refusal +} + func FetchContactedDaemon(err error, proxied bool) bool { if err == nil { return true diff --git a/schema/trajectory-sidecar.schema.json b/schema/trajectory-sidecar.schema.json index 1d46665..5b5e620 100644 --- a/schema/trajectory-sidecar.schema.json +++ b/schema/trajectory-sidecar.schema.json @@ -81,6 +81,27 @@ "description": "Bare UUID of the immediate parent cascade." }, "completeness": { + "description": "Verified projection evidence or a provisional capture marked unverified.", + "if": { + "required": [ + "state" + ] + }, + "then": { + "required": [ + "reason" + ], + "properties": { + "state": { + "const": "unverified" + }, + "reason": { + "type": "string", + "minLength": 1 + } + }, + "additionalProperties": false + }, "type": "object", "additionalProperties": true } diff --git a/scripts/test_apply_overlays.py b/scripts/test_apply_overlays.py index 3c898b9..fec2a36 100644 --- a/scripts/test_apply_overlays.py +++ b/scripts/test_apply_overlays.py @@ -40,6 +40,7 @@ "generate_tools158", "generate_workflow159", "generate_schedule1510", + "generate_policy1510", ) diff --git a/scripts/test_generate_config150.py b/scripts/test_generate_config150.py index 3c2bc4f..c1d428a 100644 --- a/scripts/test_generate_config150.py +++ b/scripts/test_generate_config150.py @@ -28,6 +28,7 @@ def test_exact_overlay_and_reproduction(self): import generate_tools158 import generate_workflow159 import generate_schedule1510 + import generate_policy1510 augmented = generate_config152.augment(result) previous = Path(directory) / 'config152.gz' previous.write_bytes(gzip.compress(augmented.SerializeToString(deterministic=True), mtime=0)) @@ -37,7 +38,7 @@ def test_exact_overlay_and_reproduction(self): current = descriptor_pb2.FileDescriptorSet.FromString(gzip.decompress(output.read_bytes())) rebuilt = current for overlay in (generate_config157, generate_planexecute157, generate_tools158, - generate_workflow159, generate_schedule1510): + generate_workflow159, generate_schedule1510, generate_policy1510): rebuilt = overlay.augment(rebuilt) self.assertEqual(rebuilt.SerializeToString(deterministic=True), gzip.decompress( (REPO / 'internal/completeness/schema.pb.gz').read_bytes())) diff --git a/scripts/test_generate_planexecute157.py b/scripts/test_generate_planexecute157.py index 60862b5..e22dfad 100644 --- a/scripts/test_generate_planexecute157.py +++ b/scripts/test_generate_planexecute157.py @@ -12,6 +12,7 @@ import generate_tools158 import generate_workflow159 import generate_schedule1510 + import generate_policy1510 def strip_planexecute(bundle): @@ -24,6 +25,8 @@ def strip_planexecute(bundle): def strip_tools158(bundle): + if any(field.name == "deployment" for field in generate_policy1510.policy(bundle).field): + generate_policy1510.strip(bundle) tools = generator.helpers.one(bundle.file, generate_tools158.TOOLS_FILE) if generate_schedule1510.TIMESTAMP in tools.dependency: generate_schedule1510.strip(bundle) diff --git a/scripts/test_generate_policy1510.py b/scripts/test_generate_policy1510.py new file mode 100644 index 0000000..03c6259 --- /dev/null +++ b/scripts/test_generate_policy1510.py @@ -0,0 +1,31 @@ +"""Verify the authenticated policy fields and restore the exact previous bundle.""" +import gzip +import hashlib +import unittest + +from test_generate_parentmsgs1410 import descriptor_pb2, REPO + +if descriptor_pb2 is not None: + import generate_policy1510 as generator + + +@unittest.skipIf(descriptor_pb2 is None, "google.protobuf is not installed") +class Policy1510(unittest.TestCase): + def test_authenticated_delta_and_round_trip(self): + production = gzip.decompress((REPO / "internal/completeness/schema.pb.gz").read_bytes()) + result = descriptor_pb2.FileDescriptorSet.FromString(production) + source = descriptor_pb2.FileDescriptorSet.FromString(generator.SNAPSHOT.read_bytes()) + source.DiscardUnknownFields() + target, original = (generator.policy(bundle) for bundle in (result, source)) + for name, _, _, present in generator.FIELDS: + actual = generator.helpers.one(target.field, name) + expected = generator.helpers.one(original.field, name) + if present: + self.assertEqual(target.oneof_decl[actual.oneof_index], original.oneof_decl[expected.oneof_index]) + expected.oneof_index = actual.oneof_index + self.assertEqual(actual, expected) + + base = generator.strip(result) + self.assertEqual(hashlib.sha256(base.SerializeToString(deterministic=True)).hexdigest(), + "f142a51963d6af1df06072e4c35d92be4a75dce1c545eeb421a7d4837b9483a2") + self.assertEqual(generator.augment(base).SerializeToString(deterministic=True), production) diff --git a/scripts/test_generate_schedule1510.py b/scripts/test_generate_schedule1510.py index 0627ad1..c50662e 100644 --- a/scripts/test_generate_schedule1510.py +++ b/scripts/test_generate_schedule1510.py @@ -9,13 +9,15 @@ if descriptor_pb2 is not None: import generate_schedule1510 as generator + import generate_policy1510 @unittest.skipIf(descriptor_pb2 is None, 'google.protobuf is not installed') class Schedule1510(unittest.TestCase): def test_authenticated_delta_and_round_trip(self): production = gzip.decompress((REPO / 'internal/completeness/schema.pb.gz').read_bytes()) - result = descriptor_pb2.FileDescriptorSet.FromString(production) + result = generate_policy1510.strip(descriptor_pb2.FileDescriptorSet.FromString(production)) + production = result.SerializeToString(deterministic=True) source = descriptor_pb2.FileDescriptorSet.FromString(generator.SNAPSHOT.read_bytes()) source.DiscardUnknownFields() diff --git a/scripts/test_generate_tools158.py b/scripts/test_generate_tools158.py index e161dec..9d71241 100644 --- a/scripts/test_generate_tools158.py +++ b/scripts/test_generate_tools158.py @@ -11,6 +11,7 @@ import generate_tools158 as generator import generate_workflow159 import generate_schedule1510 + import generate_policy1510 @unittest.skipIf(descriptor_pb2 is None, 'google.protobuf is not installed') @@ -18,7 +19,7 @@ class Tools158(unittest.TestCase): def production(self): bundle = descriptor_pb2.FileDescriptorSet.FromString(gzip.decompress( (REPO / 'internal/completeness/schema.pb.gz').read_bytes())) - return generate_workflow159.strip(generate_schedule1510.strip(bundle)) + return generate_workflow159.strip(generate_schedule1510.strip(generate_policy1510.strip(bundle))) def base(self): result = generator.strip(self.production()) diff --git a/scripts/test_generate_workflow159.py b/scripts/test_generate_workflow159.py index aec7f04..f318199 100644 --- a/scripts/test_generate_workflow159.py +++ b/scripts/test_generate_workflow159.py @@ -10,6 +10,7 @@ if descriptor_pb2 is not None: import generate_workflow159 as generator import generate_schedule1510 + import generate_policy1510 @unittest.skipIf(descriptor_pb2 is None, 'google.protobuf is not installed') @@ -17,7 +18,7 @@ class Workflow159(unittest.TestCase): def production(self): bundle = descriptor_pb2.FileDescriptorSet.FromString(gzip.decompress( (REPO / 'internal/completeness/schema.pb.gz').read_bytes())) - return generate_schedule1510.strip(bundle) + return generate_schedule1510.strip(generate_policy1510.strip(bundle)) def base(self): result = generator.strip(self.production())