Skip to content

fix(judge): normalize judge context host dir for relative --output-dir - #320

Merged
zpzjzj merged 3 commits into
alibaba:mainfrom
BiBoyang:fix/judge-context-relative-path
Oct 20, 2026
Merged

zpzjzj merged 3 commits into
alibaba:mainfrom
BiBoyang:fix/judge-context-relative-path

Conversation

@BiBoyang

@BiBoyang BiBoyang commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem

With a case using agent_judge, a relative --output-dir — the form used in the official CLI reference example (docs/zh/guide/cli-reference.md: --output-dir ./reports) — makes the whole case hard-fail during judge context materialization:

judge evaluation failed: agent_judge materialize context: judge context material path escapes context dir: "final_message.txt"

No path traversal exists; the guard misfires on the relative path itself.

Root cause

judgeContextHostDir returned .../context without resolving artifactDir, so with a relative --output-dir the resulting MaterializedContext.Dir stayed relative. isWithin resolved the root with filepath.Abs but compared it against the candidate as passed; filepath.Rel(absRoot, relPath) then returns an error, isWithin reports false, and the caller surfaces a misleading escape error.

Fix

Normalize at the source: judgeContextHostDir now returns an absolute path in both branches (the non-empty artifactDir branch and the MkdirTemp branch, whose TMPDIR may itself be relative). MaterializedContext.Dir is now always absolute, so safeMaterialHostPath always receives an absolute root and downstream writes cannot depend on the process working directory.

The previous guard-side filepath.Abs on the candidate is reverted; isWithin keeps its strict lexical comparison and traversal is still rejected.

Testing

  • TestSafeMaterialHostPath: absolute root + plain/nested file exact-value assertions; ../escape.txt rejected.
  • TestMaterializeJudgeContext_RelativeArtifactDir: relative artifact dir → mc.Dir absolute, exact expected directory, final_message.txt written.
  • TestMaterializeJudgeContext_EmptyArtifactDirUsesAbsoluteTempDir: relative TMPDIR/TMP/TEMP → mc.Dir still absolute (guards the MkdirTemp branch).
  • Mutation checks: removing the Abs reproduces the original error; reverting the empty branch makes the empty-dir test fail.
  • go test ./..., go test -race ./internal/judge/, golangci-lint run ./internal/judge/..., go vet ./internal/judge/, go mod tidy -diff — all green.
  • All six release GOOS/GOARCH targets build with CGO_ENABLED=0.

Fixes #319

isWithin resolved the root to an absolute path but not the candidate, so
a relative --output-dir (the form used in the official CLI reference)
flowed down as a relative artifact dir and filepath.Rel errored out,
failing the whole case with a misleading 'escapes context dir' message.

Abs the candidate path before comparing. Regression test covers both the
relative-root acceptance and the traversal rejection.

Fixes alibaba#319
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-18T22:43:45.852426Z b99f8e5 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@BiBoyang

Copy link
Copy Markdown
Contributor Author

Extended CI's GoReleaser Check failed twice — run 39173299434 and run 39185813268 — both with "The self-hosted runner lost communication with the server" while GoReleaser snapshot build was running, on two different runners, with the job logs missing. The same job passed in ~2 min on 2026-10-16, and all other checks are green on d63dbd7 (Build & Test, Lint, E2E Smoke, both E2E runtimes, CodeQL, CI Gate).

On the second run, the runner that hosted the job had just completed Classify changes and the docker E2E seconds earlier, then stopped heartbeating about a minute into the snapshot build; GitHub declared the job failed ~10 minutes later. The release dry-run job builds six goos/goarch targets with cache: false and also runs the go mod tidy before-hook, so a cold-cache run is the heavy part of that job.

Could a maintainer re-run the failed job? If it keeps dying at this step, caching modules for the dry-run job, dropping the redundant go mod tidy hook, or limiting build parallelism would likely fix it.

Resolve the judge context host directory to an absolute path before
materialization. With a relative --output-dir, mc.Dir used to stay
relative, so isWithin compared an absolute root against a relative
candidate and reported a misleading "escapes context dir" error.

Normalize at the source instead of widening the guard: safeMaterialHostPath
now always receives an absolute root, so the returned path no longer
depends on the process working directory when the file is written. Cover
the MkdirTemp branch too, since TMPDIR itself may be relative.

Replace the vacuous relative-root assertion with exact-value table tests
and entry-point regressions for relative and empty artifact dirs. Update
the changelog wording to match the implementation.
@BiBoyang BiBoyang changed the title fix(judge): accept relative paths in judge context material guard fix(judge): normalize judge context host dir for relative --output-dir Oct 19, 2026
@BiBoyang

Copy link
Copy Markdown
Contributor Author

Updated with the revised fix at 45013ba and re-ran the full validation.

What changed since the first revision:

  • Normalize judgeContextHostDir to an absolute path in both branches instead of widening isWithin. MaterializedContext.Dir is now always absolute, so downstream writes no longer depend on the process working directory (including the case where TMPDIR itself is relative).
  • isWithin keeps its strict lexical comparison; path traversal is still rejected.
  • Replaced the vacuous !filepath.IsAbs(got) && ... assertion with exact-value table tests, plus entry-point regressions for relative and empty artifact dirs.
  • Mutation checks confirm the new tests fail if the normalization is removed.

CI is 14/14 green, including GoReleaser Check. The earlier GoReleaser failures were runner OOM, tracked separately in #323.

@zpzjzj @hittyt @JHWang-1997 could one of you take a look when you have a moment? Happy to adjust the scope or approach.

@zpzjzj zpzjzj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zpzjzj
zpzjzj added this pull request to the merge queue Oct 20, 2026
Merged via the queue into alibaba:main with commit f84b842 Oct 20, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: agent_judge hard-fails with misleading 'escapes context dir' when --output-dir is relative

2 participants