You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
coding_env: CodeSafetyTransform flags longer names that contain a dangerous pattern #1581
CodeSafetyTransform in envs/coding_env/server/transforms.py matches its dangerous patterns with plain re.search and no word boundaries, the same design #1572 fixes in julia_env. It has the same kind of false positive: ordinary code gets the safety penalty because a longer name contains a pattern.
Checked against current main with the transform's own pattern list:
code
flagged as
ast.literal_eval(s)
eval\(
urllib.request.urlopen(u)
open\(
import osmnx
import\s+os
import os_helpers
import\s+os
A plain \b on every pattern is not a drop-in fix, which is why this is an issue and not part of #1572:
open\( also catches os.popen( and subprocess.Popen( today. Those are real shell calls, and \bopen\( stops flagging them.
\beval\( still flags method calls such as model.eval() and df.eval("a + b"), because the . before eval is a boundary.
So each pattern needs its own decision. One option is per-pattern boundaries plus explicit popen\( and Popen\( entries, so nothing that is flagged today stops being flagged by accident. Another is an ast walk over call and import nodes, which tells eval(x) apart from model.eval(). The tests/envs/test_python_codeact_rewards.py cases (import os, eval('1+1'), exec('x=1'), open('f.txt')) should keep their current rewards either way.
The other copy of this pattern is envs/julia_env/server/julia_transforms.py, handled in #1572. I'm happy to send a PR for coding_env once there's a preferred direction.
\bimport\s+os\b stops matching import osmnx and import os_helpers. import os, import os.path and import os, sys still match.
(?<![\w.])eval\( skips ast.literal_eval(...) and method calls such as model.eval() and df.eval(...). A bare \beval\( would keep flagging model.eval(), which is common in PyTorch code. exec gets the same treatment.
\bopen\( skips urlopen( but still flags open(, io.open( and os.open(. The explicit popen\( and Popen\( entries keep os.popen( and subprocess.Popen( flagged, since \bopen\( would otherwise stop catching them.
I checked this against the cases in the issue and the edge cases above. Every call flagged today is still flagged except one: builtins.eval(...), which is the price of skipping .eval( method calls. A regex can be bypassed anyway (getattr(builtins, "ev" + "al")), so this seems an acceptable trade-off for a reward signal. If you'd rather keep it flagged, I'd use \beval\( and leave model.eval() as a known false positive.
I'd skip the ast walk: it adds code for a best-effort reward check, and the regex version fixes the reported cases.
Tests: the test_python_codeact_rewards.py cases keep their rewards. I'd add direct tests on CodeSafetyTransform, like TestJuliaSafetyTransform in #1572. They would cover the names that shouldn't be flagged and the calls that still should, and need no smolagents.
@zuraiz-anjum you offered to send this PR, so I'm happy to leave it to you if you still plan to. Otherwise I can open it once the direction is OK.
Thanks for picking this up, @seyhunak. Per-pattern boundaries is the direction I had in mind too, and since #1627 is already up, keep it. No need for a second PR from me.
I checked your list against main and left one note on the PR: asyncio.create_subprocess_exec( is flagged today and stops being flagged with the lookbehind, so it wants its own entry like popen(.
Activity
seyhunak commentedon Nov 14, 2026
Here's a direction that keeps the regex approach and the shape of #1572, with a boundary decision for each pattern:
\bimport\s+os\bstops matchingimport osmnxandimport os_helpers.import os,import os.pathandimport os, sysstill match.(?<![\w.])eval\(skipsast.literal_eval(...)and method calls such asmodel.eval()anddf.eval(...). A bare\beval\(would keep flaggingmodel.eval(), which is common in PyTorch code.execgets the same treatment.\bopen\(skipsurlopen(but still flagsopen(,io.open(andos.open(. The explicitpopen\(andPopen\(entries keepos.popen(andsubprocess.Popen(flagged, since\bopen\(would otherwise stop catching them.I checked this against the cases in the issue and the edge cases above. Every call flagged today is still flagged except one:
builtins.eval(...), which is the price of skipping.eval(method calls. A regex can be bypassed anyway (getattr(builtins, "ev" + "al")), so this seems an acceptable trade-off for a reward signal. If you'd rather keep it flagged, I'd use\beval\(and leavemodel.eval()as a known false positive.I'd skip the
astwalk: it adds code for a best-effort reward check, and the regex version fixes the reported cases.Tests: the
test_python_codeact_rewards.pycases keep their rewards. I'd add direct tests onCodeSafetyTransform, likeTestJuliaSafetyTransformin #1572. They would cover the names that shouldn't be flagged and the calls that still should, and need no smolagents.@zuraiz-anjum you offered to send this PR, so I'm happy to leave it to you if you still plan to. Otherwise I can open it once the direction is OK.
zuraiz-anjum commentedon Nov 14, 2026
Thanks for picking this up, @seyhunak. Per-pattern boundaries is the direction I had in mind too, and since #1627 is already up, keep it. No need for a second PR from me.
I checked your list against main and left one note on the PR:
asyncio.create_subprocess_exec(is flagged today and stops being flagged with the lookbehind, so it wants its own entry likepopen(.