Skip to content

fix(guardrails): make Terra trajectory gates explicit - #2219

Merged
gabrielavaduva merged 5 commits into
mainfrom
codex/guardrail-terra-green
Jul 24, 2026
Merged

fix(guardrails): make Terra trajectory gates explicit#2219
gabrielavaduva merged 5 commits into
mainfrom
codex/guardrail-terra-green

Conversation

@apetraru-uipath

@apetraru-uipath apetraru-uipath commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

What changed?

  • Route a named Tool plus an exact mechanical predicate to a custom
    deterministic guardrail before built-in catalog ranking. Quoted text and a
    distinct all-caps token such as CONFIDENTIAL are explicitly treated as
    literals, not broadened into an LLM-based semantic category.
  • Require the deterministic result to use the exact resource.json Tool name,
    Tool scope, the matching custom rule (word/contains for a literal), and a
    blocking action. Before validation, re-read agent.json; if a built-in was
    written, replace it with the required custom rule.
  • Make the required coded SDK documentation fetch an explicit first-step
    completion gate.
  • Make escalation lookup unconditional: after resources list, both the
    exact-match and no-match branches must attempt
    uip solution resources get before editing, rejecting, or responding.
  • Require low-code scaffolding before escalation lookup, and require
    uip agent validate after refresh before reporting conversational work
    complete.
  • Make the three coded guardrail documentation criteria agent-portable: they
    still require the exact UiPath SDK guardrail URL, but accept either Claude's
    WebFetch or Codex's Bash-based fetch.
  • Limit changes to uipath-agents guardrail references and three coded
    guardrail task definitions. No non-guardrail skill or workflow is changed.

How has this been tested?

  • 94 passedpytest tests/scripts -q
  • All six targeted task definitions resolve with coder-eval plan under
    tests/experiments/smoke.yaml.
  • Codex gpt-5.6-terra on the exact PR head (2f9ec36e5):
    6/6 passed with every weighted and individual criterion score at
    1.0
    .
    These are the six non-Maestro guardrail failures from the official
    2026-07-23_04-33-01 daily Codex baseline.
  • The literal custom-rule branch also
    passed in isolation
    on the same head, producing $guardrailType: "custom", Tool scope, the
    exact Send message to channel name, word/contains, and
    value: "CONFIDENTIAL".
  • PR smoke uses Claude Sonnet 5. On the exact head, all three guardrail smoke
    tasks passed at 1.0 (including escalation), and the reviewer threshold
    passed. The broad
    26-task aggregate
    is 23/26 because three unrelated tasks failed by timeout/max-turn exhaustion:
    coded push/pull, bindings sync, and coded Integration Service.
  • The Task Driver Gate is green after rebasing onto current main.
  • git diff origin/main --check passes.

Are there any breaking changes?

  • None
  • Yes

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Claude finished @apetraru-uipath's task in 3m 16s —— View job


PR Review: fix(guardrails): harden Codex Terra routing and validation

  • Read review criteria and project conventions
  • Analyze full diff (12 files, +1164/−33)
  • Read each changed file in full
  • Check relative links and cross-skill references
  • Check for secrets/tokens/personal paths
  • Post review findings

Summary

This PR hardens guardrail skill guidance for Codex/Terra by: (1) reordering the SDK WebFetch to be the first external operation, (2) adding explicit callable-rule recipes for deterministic coded guardrails, (3) adding a "fail closed" policy for net-new recommendations when catalog/list data is unavailable, (4) adding a guardrail routing gate to the review skill, (5) allowing the review skill to write one report file to an explicitly requested external path, and (6) replacing regex-based checker logic with AST-based analysis for deterministic guardrail validation. Includes three new regression test files and substantial checker rewrites.

Change-by-Change Review

1. docs/superpowers/plans/2026-07-23-codex-guardrail-terra-remediation.md (new)

Severity: OK

Internal planning doc. Clear constraints and task decomposition. No issues.

2. docs/superpowers/specs/2026-07-23-codex-guardrail-terra-remediation-design.md (new)

Severity: OK

Design spec matching the plan. Approach is well-scoped and evidence-driven.

3. skills/uipath-agents/references/coded/capabilities/guardrails/guardrails-recommend.md

Severity: OK

The Step 0 rewrite correctly front-loads the core SDK WebFetch before catalog/list calls and project inspection. Validate Mode now has explicit standalone Bash call instructions (never chain with &&), clear advisory vs. fail-closed semantics, and the Critical Rules reflect the new ordering. The duplication between Validate Mode body and Critical Rules 1–2 is within acceptable bounds — both are authoritative entry points agents may load independently.

One observation: the Validate Mode section duplicates the catalog/list CLI commands with fenced bash blocks (lines 248–253 in the diff), which the Recommend Mode section doesn't — Recommend Mode relies on the Step 0 blocks. This is intentional (agents may jump to Validate Mode directly), but worth noting for future maintenance.

4. skills/uipath-agents/references/coded/capabilities/guardrails/guardrails.md

Severity: OK

Two new subsections added cleanly: "Deterministic Tool middleware — callable rule" and a decorator-style CustomValidator equivalent. Both include correct code examples with lambda tool_input: and GuardrailExecutionStage.PRE. The "Do not pass JSON/dict rule objects" warning directly addresses the root cause of the coded deterministic failure. No issues.

5. skills/uipath-agents/references/lowcode/capabilities/guardrails/guardrails-recommend.md

Severity: OK

Step 0 now has explicit preflight isolation instructions. The new Step 2A ("Exact Named-Tool Deterministic Rule") correctly prioritizes literal-word custom rules over catalog-driven recommendation for exact mechanical predicates. The [guardrails.md](guardrails.md) link resolves correctly.

6. skills/uipath-review/SKILL.md

Severity: Medium — see findings below

The review skill now allows Write in allowed-tools and relaxes Critical Rule 1 from "NEVER modify any files" to "NEVER modify files inside a reviewed project or solution" with an exception for writing one external report. The guardrail routing gate (lines 259–291 in the file) is well-structured with explicit cue list and reference pointers.

Finding 6a (skills/uipath-review/SKILL.md): The allowed-tools now includes Write, but the description still says "Does NOT modify reviewed project or solution files." While technically accurate (the restriction is scoped to "reviewed" files), the expansion from Read-only to Read+Write is a meaningful permission surface change. This is intentional and documented — the report-writing exception in Critical Rule 1 explains it. Noting for reviewer awareness.

Finding 6b (skills/uipath-review/SKILL.md:259): The guardrail routing gate references two files:

  • references/agents/guardrails/guardrails-review.md — exists ✅
  • references/agents/guardrails/coded-guardrails-review.md — exists ✅

Both links verified.

7. skills/uipath-review/references/review-workflow-guide.md

Severity: OK

Report format spec updated consistently with the SKILL.md changes — same "outside every reviewed root" language, same "Do not create unsolicited files" constraint. No duplication beyond the necessary bridge content.

8. tests/scripts/test_deterministic_guardrail_checker.py (new)

Severity: OK

Thorough regression test suite for the AST-based checker. Tests cover: callable lambda rules, named function rules, constant indirection (SECRET_WORD), CustomValidator decorator wiring, functional application, locally-scoped callables and constants, and negative cases (dict rules, docstring-only mentions, default parameter values, async functions, standalone CustomValidator without @guardrail). Good coverage of edge cases.

9. tests/scripts/test_guardrail_skill_contracts.py (new)

Severity: OK

Focused content contracts that verify guardrail skill documentation contains specific phrases and ordering guarantees. Tests are deterministic (string matching, not behavior), which is appropriate for documentation contract enforcement. The _section / _squash helpers are clean.

10. tests/scripts/test_lowcode_recommend_scoped_checker.py (new)

Severity: OK

Tests the low-code scoped checker with parametrized (operator, value) pairs. Good positive/negative coverage for contains and matchesRegex operators. The ("equals", "CONFIDENTIAL") negative case correctly verifies that equals is not accepted.

11. tests/tasks/uipath-agents/coded/guardrails/deterministic/check_deterministic.py

Severity: OK

Major rewrite from regex to AST-based analysis. The _ConfiguredRuleCollector class correctly handles scope tracking, decorator visitors, and UiPathDeterministicGuardrailMiddleware extraction. The _SecretBodyVisitor properly skips nested lambdas/functions (different callables) and resolves string constants through scope chains.

The has_secret_callable function correctly:

  • Handles inline lambdas, named references, and FunctionDef nodes
  • Resolves callable names through lexical scope chains
  • Checks only the body (not docstrings, not defaults, not decorators)
  • Returns False on SyntaxError (graceful degradation)

One minor note: Scope = ast.Module | ast.FunctionDef | ast.AsyncFunctionDef (line 42) uses union type syntax that requires Python 3.10+. This is consistent with the str | None type hints elsewhere in the file and is fine for the test environment.

12. tests/tasks/uipath-agents/lowcode/guardrails/recommend_scoped/check_recommend_scoped.py

Severity: OK

Clean additions: blocks_forbidden_term() function, action type tightened from any $actionType to specifically "block", and a new check that at least one word rule can semantically match CONFIDENTIAL. The re.error handling for invalid regex patterns is correct.

What's Missing

  • Test execution evidence: The PR description claims all tests pass locally but the PR does not include a CI run link or test output artifact for the new tests/scripts/ pytest files. These tests should be verified in CI.
  • Nothing else identified — the changes are well-scoped, all relative links resolve, no CODEOWNERS changes are needed (no new skill paths), and no cross-skill dependencies are introduced.

Area Ratings

Area Status Notes
Frontmatter OK uipath-review description updated consistently with functional change
E2E Tests OK Existing task checkers updated + 3 new regression test files added
Skill Body OK Guardrail routing gate, report-writing exception, and fail-closed semantics well-structured
References & Assets OK All relative links resolve; guardrails.md and guardrails-recommend.md changes are internally consistent
Repo Hygiene OK No secrets, no cross-skill deps, no unrelated drive-by changes, docs/superpowers correctly scoped

Issues for Manual Review

  1. Semantic correctness of fail-closed vs. advisory exception: The distinction between "Recommend mode must fail closed" and "Validate mode catalog/list failures are advisory" is a domain-specific design decision. Verify this matches the intended product behavior — particularly that GuardrailCatalogUnavailable should indeed not block fixing an existing misplaced validator.
  2. Review skill Write permission: The allowed-tools now includes Write. While the guardrails (only write one report to an explicitly requested external path) are clearly documented, this does expand the skill's blast radius. Verify that the evaluation framework accounts for this.
  3. AST checker Python version: The checker uses ast.Module | ast.FunctionDef union syntax (Python 3.10+). Verify the CI test runners use Python 3.10+.

Conclusion

This is a focused, well-structured remediation PR. The skill documentation changes are internally consistent, the test coverage is thorough, and the AST-based checker is a significant quality improvement over the previous regex approach. No critical or high-severity issues found. The medium-severity note on the Write permission is intentional and well-documented. Approve.

@apetraru-uipath
apetraru-uipath force-pushed the codex/guardrail-terra-green branch from bda1fc8 to 7d14791 Compare July 23, 2026 09:06
@apetraru-uipath
apetraru-uipath force-pushed the codex/guardrail-terra-green branch from 71a422f to 0725aad Compare July 23, 2026 10:24
@apetraru-uipath
apetraru-uipath force-pushed the codex/guardrail-terra-green branch from 7ee0240 to 59f5dc1 Compare July 23, 2026 12:41
@apetraru-uipath
apetraru-uipath force-pushed the codex/guardrail-terra-green branch 3 times, most recently from 4a1a015 to 0a29b10 Compare July 23, 2026 20:41
@apetraru-uipath apetraru-uipath changed the title fix(guardrails): harden Codex Terra routing and validation fix(guardrails): make Terra trajectory gates explicit Jul 23, 2026
@apetraru-uipath
apetraru-uipath force-pushed the codex/guardrail-terra-green branch from ff380e3 to 51689e2 Compare July 24, 2026 03:45
@apetraru-uipath
apetraru-uipath force-pushed the codex/guardrail-terra-green branch from a6b6f53 to 340aa63 Compare July 24, 2026 05:53
description: "Agent fetched the UiPath Python SDK guardrail docs"
tool_name: "WebFetch"
command_pattern: 'uipath\.github\.io/uipath-python/.*guardrails'
min_count: 1

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed tool_name: "WebFetch" because it made the documentation-fetch criterion Claude-specific.
Claude Sonnet fetches the SDK page with WebFetch, while Codex/Terra typically uses Bash with curl. The exact UiPath SDK URL remains mandatory through command_pattern, so the test still verifies the required documentation step—it just accepts either transport.
Without this change, Codex could fetch the correct page and perform the task correctly but still receive a false failure solely because it didn’t use Claude’s tool name. The tradeoff is that any tool matching the exact URL is accepted, which is intentional for cross-agent tests.

@gabrielavaduva
gabrielavaduva merged commit a0f3535 into main Jul 24, 2026
30 of 33 checks passed
@gabrielavaduva
gabrielavaduva deleted the codex/guardrail-terra-green branch July 24, 2026 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants