Record the DeepSWE OpenHuman smoke evaluation

docs/experiments/2026-09-18-deepswe-openhuman-smoke.md

@@ -0,0 +1,120 @@
+# DeepSWE OpenHuman smoke
+
+**Date:** 2026-09-18
+
+**Status:** Recorded
+
+**Code:** `deepswe_hive` on `issue-57-deepswe-eval`, default model
+`openai/gpt-oss-120b:nitro`, 24 committed-turn cap
+
+**Spec:** [issue 57](https://github.com/tinyhumansai/tinyhivemind/issues/57)
+
+**Decision:** Do not treat the local acceptance fixture as benchmark evidence;
+fix bounded-run finalization before expanding the DeepSWE slice.
+
+## Question
+
+Can the OpenHuman-backed four-seat hive produce canonically correct patches on
+a small real DeepSWE v1.1 slice while every agent action remains inside an
+offline Docker workspace?
+
+## Corpus and slice
+
+The gated official dataset could not be fetched without Hugging Face
+credentials. This run used the public third-party materialization
+`luolc/deep-swe-1-1-materialized`, which identifies its source as
+`datacurve-ai/deep-swe` commit
+`435ee89ec2f2e2289f33b0da4f992f0b7b7266b9`. The materialization contains 113
+tasks. Its parquet SHA-256 was verified as
+`954184ffb6fd88c798171dfc8577793b29631bff35c8d02f6fa4bbf88a44abd0`.
+
+This was a three-task smoke, not a score over the full 113-task benchmark. The
+slice was the first three task IDs in lexical order, spanning two upstream
+repositories:
+
+| task | repository | base commit |
+| --- | --- | --- |
+| `abs-module-cache-flags` | `abs-lang/abs` | `cb1b3b671d0ee9fa9da9f7b02f86967953ffd10a` |
+| `abs-stepped-slices` | `abs-lang/abs` | `cb1b3b671d0ee9fa9da9f7b02f86967953ffd10a` |
+| `actionlint-action-pinning-lint` | `rhysd/actionlint` | `0bdc95715fa58f64e3fd6e63b0f89be8733cbbab` |
+
+Each checkout was extracted from its task image and verified at the stated
+commit with no tracked, untracked, ignored, or gitlink entries. Focused public
+baseline tests passed before the run.
+
+## Isolation
+
+Agent actions ran as the checkout owner in ephemeral containers with Docker
+network mode `none`, a cleared environment, all capabilities dropped,
+`no-new-privileges`, bounded memory/CPU/PIDs, and `.git` masked. Agents received
+only repository read/write/edit/shell/test MCP tools plus the two hive actions.
+They had no browser, web search, fetch, host shell, credentials, dataset,
+reference solution, or hidden-verifier mount. Provider HTTP ran in the host
+process, outside the agent containers.
+
+The task images stored their preloaded Go modules below `/root/go`. Derived
+agent images changed only directory/module-cache read permissions so the
+unprivileged checkout owner could run tests offline. Canonical grading used the
+unaltered task images in fresh, networkless containers.
+
+## Results
+
+The official task reward is binary. All three captured patches were empty, and
+all three canonical graders returned zero:
+
+| task | wall time | runner termination | patch | F2P | P2P | partial diagnostic | reward |
+| --- | ---: | --- | ---: | ---: | ---: | ---: | ---: |
+| `abs-module-cache-flags` | 2:21.79 | next round of 4 exceeded cap | 0 B | 0/20 | 3/3 | 0.1304 | **0** |
+| `abs-stepped-slices` | 8:40.23 | next round of 3 exceeded cap | 0 B | 0/6 | 6/6 | 0.5000 | **0** |
+| `actionlint-action-pinning-lint` | 1:05.79 | next round of 3 exceeded cap | 0 B | 0/55 | 145/145 | 0.7250 | **0** |
+
+**Smoke score: 0/3 (0%).** The partial column is grader diagnostics, not the
+DeepSWE reward and not an alternative score.
+
+Provider-reported usage was:
+
+| task | billed requests | input tokens | output tokens | cached input subset | cost |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| `abs-module-cache-flags` | 120 | 1,087,847 | 25,059 | 615,936 | $1.976399 |
+| `abs-stepped-slices` | 162 | 2,379,696 | 29,732 | 1,095,936 | $4.626041 |
+| `actionlint-action-pinning-lint` | 125 | 1,452,767 | 21,953 | 759,808 | $2.636114 |
+| **total** | **407** | **4,920,310** | **76,744** | **2,471,680** | **$9.238554** |
+
+The cached count is a subset of input tokens, not additional usage. Every cost
+record identified its source as `provider_charged`.
+
+The runner rejected a pending round when it would cross the 24-turn cap. It
+exited before serializing `result.json` or the committed-turn counter, so an
+exact per-task committed-turn count is unavailable. Raw evidence establishes
+only that each run stopped below or at 24 turns before its next three- or
+four-seat round. Archived OpenHuman session snapshots numbered 29, 35, and 29;
+these include protocol retries and must not be reported as committed turns.
+
+The local one-task acceptance fixture passed 1/1 during runner validation. It
+is a synthetic, visible-test fixture and is not benchmark evidence; the real
+smoke result above supersedes it for any claim about DeepSWE performance.
+
+## What happened
+
+The seats repeatedly announced completion without making source changes. The
+reviewer and tester sometimes reported success despite finding no implementation
+or seeing network-blocked dependency attempts. The completion protocol did not
+reach quorum before the cap, and the strict round-width check then terminated
+the run. This is a useful failure: the harness prevented unbounded calls and
+the canonical hidden tests rejected the empty work.
+
+One Task 2 shell command recursively searched the task image until its
+600-second timeout. Because the Docker action timeout and model-turn timeout
+were identical, the MCP process could be stopped before cleanup and left one
+exited container. It was removed, and the runner now uses nested 540-second
+command and 570-second action deadlines inside the unchanged 600-second turn
+deadline. The final Docker audit found no `deepswe-*` containers.
+
+## Consequences
+
+Before a larger slice, the runner should write a terminal result at budget
+exhaustion, including committed turns, retry counts, and the captured patch,
+rather than exiting before serialization. The agent protocol also needs a
+grounded-progress check: completion claims without a non-empty diff or relevant
+test evidence should not advance the episode. Neither change alters this
+recorded score.

docs/experiments/2026-09-20-deepswe-openhuman-main-rerun.md

@@ -0,0 +1,75 @@
+# DeepSWE OpenHuman upstream-main rerun
+
+**Date:** 2026-09-20
+
+**Status:** Recorded
+
+**Code:** `deepswe_hive` with OpenHuman
+`b8eeec0b8d1254a516a9f06477823b74ee7ed62c`, default model
+`openai/gpt-oss-120b:nitro`, 24 committed-turn cap
+
+**Spec:** [issue 57](https://github.com/tinyhumansai/tinyhivemind/issues/57)
+
+## Question
+
+Does updating the OpenHuman dependency to its 2026-09-20 `upstream/main`
+revision improve the three-task Docker-isolated smoke, and what fraction of
+provider input tokens are served from cache?
+
+## Method
+
+This reran the same three tasks, base commits, model, public baselines, task
+images, and hidden verifiers as the 2026-09-18 smoke. Each checkout was freshly
+extracted from its task image and verified clean at its exact base commit.
+Focused public baseline tests passed inside the derived agent image before the
+run.
+
+Agent actions ran as the checkout owner in ephemeral Docker containers with
+network mode `none`; provider HTTP remained in the host process. Agents had no
+browser, search, host shell, credentials, reference solution, dataset, or
+hidden-verifier mount. Canonical grading ran later in separate networkless
+containers using the unaltered official task images.
+
+OpenHuman `b8eeec0` requires an embedded agent's explicit definition to also
+be present in its config-backed catalogue during hosted turn preparation. The
+runner now registers that equivalent catalogue entry. The same revision also
+sanitizes hosted provider failures, so an error without a retained retryability
+classification fails closed rather than being guessed retryable. All 45 runner
+tests passed before the paid rerun.
+
+## Results
+
+| task | wall time | runner termination | patch | F2P | P2P | partial diagnostic | reward |
+| --- | ---: | --- | ---: | ---: | ---: | ---: | ---: |
+| `abs-module-cache-flags` | 10:44.98 | implementer emitted zero actions after 3 attempts | 0 B | 0/20 | 3/3 | 0.1304 | **0** |
+| `abs-stepped-slices` | 1:14.47 | tester emitted zero actions after 3 attempts | 0 B | 0/6 | 6/6 | 0.5000 | **0** |
+| `actionlint-action-pinning-lint` | 0:59.22 | episode exceeded 24 turns | 216 B | 0/55 | 0/145 | 0.0000 | **0** |
+
+**Smoke score: 0/3 (0%).** The partial column is grader diagnostics, not an
+alternative DeepSWE score. The third patch added only a three-line placeholder
+file, and the hidden suite did not build; the other two runs produced no patch.
+
+Provider-reported usage was:
+
+| task | charged requests | input tokens | cached input subset | cache hit rate | output tokens | cost |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| `abs-module-cache-flags` | 159 | 1,529,931 | 907,392 | 59.31% | 33,262 | $2.638765 |
+| `abs-stepped-slices` | 94 | 1,073,954 | 441,088 | 41.07% | 15,876 | $2.269064 |
+| `actionlint-action-pinning-lint` | 140 | 1,634,797 | 798,336 | 48.83% | 26,171 | $3.141449 |
+| **total** | **393** | **4,238,682** | **2,146,816** | **50.65%** | **75,309** | **$8.049278** |
+
+OpenHuman also wrote one zero-cost `estimated` mirror for every charged record;
+the table counts only `provider_charged` records so calls and tokens are not
+double-counted. Cache hit rate is
+`cached_input_tokens / input_tokens`; cached input is a subset of input, not
+additional usage. The aggregate rate was 50.6482%, about 0.42 percentage points
+higher than the original smoke's 50.23%.
+
+## Observation
+
+The dependency update did not improve canonical correctness. Agents spent more
+input tokens than in the first smoke, but two rounds failed the native-action
+protocol and the only captured patch was a placeholder. One Task 1 action also
+ran a recursive grep outside `/workspace` within the container image until the
+existing 540-second command timeout stopped it. The container still had no
+network or host/verifier mount and was removed by cleanup.

docs/experiments/README.md

@@ -38,6 +38,7 @@ does.
 | [2026-09-09](2026-09-09-desk-lessons.md) | What should be built next after PE 1006? | Working notes and measurements from runs 21–27 |
 | [2026-09-08](2026-09-08-pe1006-tool-room.md) | Does a tool-call room with a standing account beat a fenced one? | PE 1006 solved; re-reading fell to 3% of calls, but no fold ever fired |
 | [2026-09-17](2026-09-17-jev-decision-evaluation.md) | What does Jev buy and cost against GPT-5-mini strict JSON on typed hive decisions? | 43–55× lower tail latency and 99.2% lower estimated cost, with an 11.45-point accuracy tradeoff concentrated in evidence scoring |
+| [2026-09-18](2026-09-18-deepswe-openhuman-smoke.md) | Can the offline OpenHuman hive solve a real three-task DeepSWE smoke? | No: 0/3; every run exhausted the bounded round budget with an empty patch |
 
 ## Reading order
 

examples/openhuman/README.md

@@ -109,10 +109,12 @@ cannot resume stale state and no preexisting outbox child can be reused.
 
 Every agent file read/write/edit and shell/test call uses a fresh container
 with `--network none`, 1 GiB memory, 2 CPUs, 256 PIDs, dropped capabilities,
-no-new-privileges, an `env -i` process environment, and the checkout at
-`/workspace`. A read-only empty mount covers `/workspace/.git`, including when
-the checkout's `.git` is a worktree pointer, so agent commands cannot reach or
-mutate the source history. Model-supplied file content is limited to exactly
+no-new-privileges, the host process's numeric UID/GID, an `env -i` process
+environment, and the checkout at `/workspace`. A read-only empty mount covers
+`/workspace/.git`, including when the checkout's `.git` is a worktree pointer,
+so agent commands cannot reach or mutate the source history. The file tools
+also reject targets whose resolved path leaves `/workspace`, including through
+a symlink. Model-supplied file content is limited to exactly
 1 MiB (1,048,576 bytes), staged before Docker starts in a host-owned temporary
 file outside the checkout, and mounted read-only at `/tmp/deepswe-input`; the
 fixed container wrapper consumes that path, so Docker receives no agent-chosen

examples/openhuman/src/bin/deepswe_hive/README.md

@@ -10,9 +10,11 @@
 | `test.rs` | Exercises parsing, refusal paths, confinement arguments, patch capture, and output shape. |
 
 The model provider remains in the host process. Agent file and shell tools run
-only inside the configured Docker image, with the disposable checkout mounted
-at `/workspace`, its real `.git` hidden by a read-only empty mount, and no
-provider credential copied into the cleared action environment. The runner
+only inside the configured Docker image as the host process's numeric UID/GID,
+with the disposable checkout mounted at `/workspace`, its real `.git` hidden by
+a read-only empty mount, and no provider credential copied into the cleared
+action environment. File read, write, and edit targets are resolved inside the
+container and rejected when a symlink would leave `/workspace`. The runner
 does not perform destructive Git operations, but an agent can modify or delete
 files inside the disposable workspace it was explicitly given.
 

examples/openhuman/src/bin/deepswe_hive/sandbox.rs

@@ -25,7 +25,10 @@ pub(super) use preflight::preflight_with_limits;
 pub(super) use test_support::shell_with_limits_at;
 
 const DEFAULT_IMAGE: &str = "tinyhivemind-deepswe:local";
-const COMMAND_TIMEOUT_SECONDS: &str = "600";
+// Keep both Docker deadlines inside the 600-second model-turn deadline so the
+// MCP server retains time to remove a timed-out container before it is stopped.
+const COMMAND_TIMEOUT_SECONDS: &str = "540";
+const ACTION_TIMEOUT: Duration = Duration::from_secs(570);
 pub(super) const INSPECTOR_TIMEOUT: Duration = Duration::from_secs(600);
 pub(super) const MAX_PATCH_BYTES: u64 = 32 * 1024 * 1024;
 pub(super) const MAX_ACTION_OUTPUT_BYTES: u64 = 1024 * 1024;
@@ -125,6 +128,7 @@ pub(super) struct DockerSandbox {
     config: SandboxConfig,
     mask: Arc<tempfile::TempDir>,
     git: GitLayout,
+    user: String,
 }
 
 #[derive(Debug)]
@@ -145,7 +149,13 @@ impl DockerSandbox {
     pub(super) fn file_read(&self, path: &str) -> anyhow::Result<String> {
         validate_relative(path)?;
         let output = self.action(
-            ["sh", "-c", "cat -- \"/workspace/$1\"", "deepswe-read", path],
+            [
+                "sh",
+                "-c",
+                "target=/workspace/$1; resolved=$(realpath -m -- \"$target\") || exit; case \"$resolved\" in /workspace/*) cat -- \"$target\" ;; *) exit 1 ;; esac",
+                "deepswe-read",
+                path,
+            ],
             &[],
         )?;
         require_success(output, "read file").map(|output| output.stdout)
@@ -157,7 +167,7 @@ impl DockerSandbox {
             [
                 "sh",
                 "-c",
-                "target=/workspace/$1; mkdir -p -- \"$(dirname -- \"$target\")\" && cat > \"$target\"",
+                "target=/workspace/$1; resolved=$(realpath -m -- \"$target\") || exit; case \"$resolved\" in /workspace/*) mkdir -p -- \"$(dirname -- \"$target\")\" && cat > \"$target\" ;; *) exit 1 ;; esac",
                 "deepswe-write",
                 path,
             ],
@@ -254,7 +264,7 @@ exit "${statuses[0]}"
     }
 
     fn action<const N: usize>(&self, args: [&str; N], input: &[u8]) -> anyhow::Result<ShellOutput> {
-        self.action_with_limits(args, input, INSPECTOR_TIMEOUT, MAX_ACTION_OUTPUT_BYTES)
+        self.action_with_limits(args, input, ACTION_TIMEOUT, MAX_ACTION_OUTPUT_BYTES)
     }
 
     fn action_with_limits<const N: usize>(
@@ -328,6 +338,7 @@ fn mask_source(sandbox: &DockerSandbox) -> PathBuf {
 fn action_args(sandbox: &DockerSandbox, create: bool) -> anyhow::Result<Vec<String>> {
     Ok(container_args(
         &sandbox.config,
+        &sandbox.user,
         false,
         [
             mount(&sandbox.config.repo_path, "/workspace", true)?,
@@ -361,11 +372,18 @@ fn inspector_args(sandbox: &DockerSandbox) -> anyhow::Result<Vec<String>> {
             false,
         )?);
     }
-    Ok(container_args(&sandbox.config, false, mounts, false))
+    Ok(container_args(
+        &sandbox.config,
+        &sandbox.user,
+        false,
+        mounts,
+        false,
+    ))
 }
 
 fn container_args(
     config: &SandboxConfig,
+    user: &str,
     remove: bool,
     mounts: impl IntoIterator<Item = String>,
     create: bool,
@@ -387,6 +405,8 @@ fn container_args(
         "ALL".into(),
         "--security-opt".into(),
         "no-new-privileges".into(),
+        "--user".into(),
+        user.into(),
     ]);
     for value in mounts {
         args.extend(["--mount".into(), value]);

examples/openhuman/src/bin/deepswe_hive/sandbox/preflight.rs

@@ -1,6 +1,7 @@
 //! Bounded Docker daemon and sandbox readiness checks.
 
 use std::io::Read;
+use std::os::unix::fs::MetadataExt;
 use std::process::{ExitStatus, Stdio};
 use std::sync::Arc;
 use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
@@ -91,7 +92,13 @@ fn preflight(
 
     let mask = Arc::new(tempfile::tempdir()?);
     prepare_mask(&mask, git.dot_git_is_file)?;
-    let sandbox = DockerSandbox { config, mask, git };
+    let owner = std::fs::metadata(mask.path())?;
+    let sandbox = DockerSandbox {
+        config,
+        mask,
+        git,
+        user: format!("{}:{}", owner.uid(), owner.gid()),
+    };
     let control = tempfile::tempdir()?;
     let cidfile = control.path().join("preflight.cid");
     let nonce = std::time::SystemTime::now()

examples/openhuman/src/bin/deepswe_hive/test.rs

@@ -1,6 +1,6 @@
 //! Contract tests for DeepSWE input, output, Git, and confinement behavior.
 
-use std::os::unix::fs::PermissionsExt;
+use std::os::unix::fs::{MetadataExt, PermissionsExt};
 use std::path::Path;
 use std::process::Command;
 use std::time::{Duration, Instant};
@@ -470,6 +470,40 @@ fn inspector_timeout_kills_a_hung_docker_cli() {
     assert!(started.elapsed() < Duration::from_secs(15));
 }
 
+#[test]
+fn sandbox_runs_as_the_checkout_owner() {
+    let (directory, task) = fixture();
+    let docker = directory.path().join("fake-docker-owner");
+    let removed = directory.path().join("fake-container-removed");
+    std::fs::write(
+        &docker,
+        format!(
+            "#!/bin/sh\ncase \"$1\" in\nversion) printf fixture ;;\ncreate) rm -f '{}'; printf abcdef1234567890 ;;\ninspect) if test -e '{}'; then printf 'Error: No such object: %s' \"$2\" >&2; exit 1; fi; printf '[{{\"HostConfig\":{{\"NetworkMode\":\"none\"}},\"Mounts\":[{{\"Destination\":\"/workspace\",\"RW\":true}},{{\"Destination\":\"/workspace/.git\",\"RW\":false}}]}}]' ;;\nstart) printf sandbox-ready ;;\nrm) touch '{}' ;;\n*) exit 1 ;;\nesac\n",
+            removed.display(),
+            removed.display(),
+            removed.display(),
+        ),
+    )
+    .expect("fake Docker");
+    let mut permissions = std::fs::metadata(&docker).expect("metadata").permissions();
+    permissions.set_mode(0o755);
+    std::fs::set_permissions(&docker, permissions).expect("executable fake Docker");
+    let sandbox = DockerSandbox::preflight(SandboxConfig {
+        repo_path: task.repo_path.clone(),
+        image: "local-fixture".into(),
+        docker,
+    })
+    .expect("preflight");
+    let metadata = std::fs::metadata(&task.repo_path).expect("checkout metadata");
+    let expected = format!("{}:{}", metadata.uid(), metadata.gid());
+    let args = action_arguments(&sandbox).expect("action arguments");
+
+    assert!(
+        args.windows(2)
+            .any(|pair| pair == ["--user", expected.as_str()])
+    );
+}
+
 #[test]
 fn live_real_docker_fixture_flow_confines_actions_and_captures_new_files() {
     if std::env::var_os("DEEPSWE_REAL_DOCKER_TEST").is_none() {