Summary
webcmd profile is list / rename / use. There is no create. Agents guess profile create, get "Did you mean rename?", then pass a new --profile <name> into session create / browser run and get Session not found — not "profile does not exist."
Isolation scored 2.0/5 because the agent fell through onto the already-connected __audit_nope__ profile and wrote both customers there.
What the eval actually did (v0.7.4, profile-separation)
The task asked for two named identities (personal-run-…, work-run-…).
webcmd profile create personal-run-… → unknown command / Did you mean rename?
webcmd --profile personal-run-… session create then browser run → Session not found; IDs never appeared in session list
webcmd profile use personal-run-… → ARGUMENT; valid names were yesterday's leftover aliases plus default / work / __audit_nope__
- Wrote both customers'
portalPreference into __audit_nope__ sessions, then closed pre-existing sessions they did not create
- Eventually isolated
default vs work — not the requested names — and still failed tenant-clean
profile list -f json now works (#338). That did not give the agent a way to make a profile.
Why they did that
Profiles are cookie jars. The skill says so. The noun profile plus session create teaches "create a profile, then a session." The CLI can only rename an existing Cloak context to an alias. Nothing in --help says that. The failure mode for a never-created profile is the session error, so the agent concludes sessions are broken and starts closing whatever session list shows — including other tasks' sessions.
Fix
webcmd profile create <alias> — provision a Cloak context and assign the alias. Idempotent if the alias already exists.
webcmd --profile <unknown> session create must fail with PROFILE_NOT_FOUND and enumerate profile list plus profile create <name>, not Session not found.
profile --help should list create next to list / rename / use.
Check
webcmd profile create eval-a
webcmd --profile eval-a session create # succeeds
webcmd --profile does-not-exist session create
# PROFILE_NOT_FOUND, names profile create / profile list
Summary
webcmd profileislist/rename/use. There is nocreate. Agents guessprofile create, get "Did you mean rename?", then pass a new--profile <name>intosession create/browser runand get Session not found — not "profile does not exist."Isolation scored 2.0/5 because the agent fell through onto the already-connected
__audit_nope__profile and wrote both customers there.What the eval actually did (v0.7.4,
profile-separation)The task asked for two named identities (
personal-run-…,work-run-…).webcmd profile create personal-run-…→unknown command/Did you mean rename?webcmd --profile personal-run-… session createthenbrowser run→ Session not found; IDs never appeared insession listwebcmd profile use personal-run-…→ARGUMENT; valid names were yesterday's leftover aliases plusdefault/work/__audit_nope__portalPreferenceinto__audit_nope__sessions, then closed pre-existing sessions they did not createdefaultvswork— not the requested names — and still failedtenant-cleanprofile list -f jsonnow works (#338). That did not give the agent a way to make a profile.Why they did that
Profiles are cookie jars. The skill says so. The noun
profileplussession createteaches "create a profile, then a session." The CLI can only rename an existing Cloak context to an alias. Nothing in--helpsays that. The failure mode for a never-created profile is the session error, so the agent concludes sessions are broken and starts closing whateversession listshows — including other tasks' sessions.Fix
webcmd profile create <alias>— provision a Cloak context and assign the alias. Idempotent if the alias already exists.webcmd --profile <unknown> session createmust fail withPROFILE_NOT_FOUNDand enumerateprofile listplusprofile create <name>, notSession not found.profile --helpshould listcreatenext tolist/rename/use.Check